Page MenuHomePhabricator

RFC: DurableAddressStorage and EphemeralAddressStorage
Changes PlannedPublic

Authored by yuhan on Oct 19 2020, 7:19 AM.



prototype only - don't plan to land this until we figure out if an address storage is necessary (i.e. it can provide reasonable user values). while this diff is open for comments and discussions, im working on improving asset store.

I believe threading here isn't in a good state. This is a working RFC diff which creates a layer of indirection between execute_step and asset_store. There are two options:

  • DurableAddressStorage: step_output_handle -> address mapping. can be configured on instance
  • EphemeralAddressStorage: default option, run scoped. it has nothing to do with the addresses and only operates on static asset metadata (execution plan)
Test Plan


Diff Detail

R1 dagster
Lint OK
No Unit Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 7:36 AM
Harbormaster failed remote builds in B19752: Diff 23973!
yuhan requested review of this revision.Oct 19 2020, 5:01 PM
yuhan retitled this revision from rfc EphemeralAddressStorage to RFC: DurableAddressStorage and EphemeralAddressStorage.Wed, Oct 28, 3:42 AM
yuhan edited the summary of this revision. (Show Details)
yuhan added reviewers: cdecarolis, sandyryza, schrockn.