HomePhabricator

mirror s3 bucket structure to local directory

Authored by prha on Tue, Oct 1, 9:05 PM.

Description

mirror s3 bucket structure to local directory

Summary:
The current default local directory structure in $DAGSTER_HOME is as follows:

$DAGSTER_HOME/
  storage/
    $run_id/
      files/ - managed files
      intermediates/ - intermediates
      compute_logs/  - compute logs
  schedules/

The current AWS s3 bucket structure is as follows:

$S3_BUCKET/
  dagster/
    runs/
      $run_id/
        files/
          managed/ - managed files
          intermediates/ - intermediates
          compute_logs/  - compute logs

This diff changes the AWS s3 bucket layout to mirror the local fs layout to the following:

$S3_BUCKET/
  dagster/
    storage/
      $run_id/
        files/ - managed files
        intermediates/ - intermediates
        compute_logs/  - compute logs

Test Plan: bk

Reviewers: #ft, max

Reviewed By: #ft, max

Differential Revision: https://dagster.phacility.com/D1142

Details

Committed
prhaWed, Oct 2, 4:30 PM
Reviewer
Restricted Project
Differential Revision
D1142: mirror s3 bucket structure to local directory
Parents
R1:414df8099e03: Add compute logging watcher process to kill tail (and itself) if parent dies
Branches
Unknown
Tags
Unknown