Page MenuHomePhabricator

[s3] cp_object support
ClosedPublic

Authored by alangenfeld on Wed, Oct 9, 11:29 PM.

Details

Reviewers
prha
Group Reviewers
Restricted Project
Commits
R1:9d2beb1f272e: [s3] cp_object support
Summary
  • add support for copy object to S3 object store
  • updates the intermediate store to be able to calculate roots based on run_id - which is necessary to provide the api of copying intermediates from a previous run
Test Plan

updated test case

Diff Detail

Repository
R1 dagster
Branch
cp_obj (branched from master)
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

alangenfeld created this revision.Wed, Oct 9, 11:29 PM
alangenfeld updated this revision to Diff 5747.Wed, Oct 9, 11:39 PM

move impl up in class heirarchy

more reliable way to get directory / path for previous run

alangenfeld edited the summary of this revision. (Show Details)Thu, Oct 10, 12:21 AM
alangenfeld added a reviewer: Restricted Project.
alangenfeld added inline comments.
python_modules/dagster/dagster/core/storage/intermediate_store.py
17–70

there may be a cleaner way if we changed the API but given copy_object_from_prev_run the IntermediateStore needs to have the function to produce a root given a run_id

alangenfeld updated this revision to Diff 5753.Thu, Oct 10, 3:24 PM
alangenfeld edited the summary of this revision. (Show Details)

rebase

prha accepted this revision.Thu, Oct 10, 3:42 PM
This revision is now accepted and ready to land.Thu, Oct 10, 3:42 PM
This revision was automatically updated to reflect the committed changes.