Page MenuHomePhabricator

mirror s3 bucket structure to local directory
ClosedPublic

Authored by prha on Tue, Oct 1, 9:28 PM.

Details

Reviewers
max
Group Reviewers
Restricted Project
Commits
R1:81f05b5021ce: mirror s3 bucket structure to local directory
Summary

The current default local directory structure in $DAGSTER_HOME is as follows:

$DAGSTER_HOME/
  storage/
    $run_id/
      files/ - managed files
      intermediates/ - intermediates
      compute_logs/  - compute logs
  schedules/

The current AWS s3 bucket structure is as follows:

$S3_BUCKET/
  dagster/
    runs/
      $run_id/
        files/
          managed/ - managed files
          intermediates/ - intermediates
        compute_logs/  - compute logs

This diff changes the AWS s3 bucket layout to mirror the local fs layout to the following:

$S3_BUCKET/
  dagster/
    storage/
      $run_id/
        files/ - managed files
        intermediates/ - intermediates
        compute_logs/  - compute logs
Test Plan

bk

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

prha created this revision.Tue, Oct 1, 9:28 PM
prha edited the summary of this revision. (Show Details)Tue, Oct 1, 9:40 PM
prha added a reviewer: Restricted Project.
max accepted this revision.Wed, Oct 2, 2:39 PM
This revision is now accepted and ready to land.Wed, Oct 2, 2:39 PM
This revision was automatically updated to reflect the committed changes.
prha edited the summary of this revision. (Show Details)Wed, Oct 2, 4:31 PM