Page MenuHomePhabricator

RFC: DurableAddressStorage and EphemeralAddressStorage
Changes PlannedPublic

Authored by yuhan on Oct 19 2020, 7:19 AM.

Details

Summary

prototype only - don't plan to land this until we figure out if an address storage is necessary (i.e. it can provide reasonable user values). while this diff is open for comments and discussions, im working on improving asset store.

I believe threading here isn't in a good state. This is a working RFC diff which creates a layer of indirection between execute_step and asset_store. There are two options:

  • DurableAddressStorage: step_output_handle -> address mapping. can be configured on instance
  • EphemeralAddressStorage: default option, run scoped. it has nothing to do with the addresses and only operates on static asset metadata (execution plan)
Test Plan

bk

Diff Detail

Repository
R1 dagster
Branch
yuhan/address-storage-ephemeral
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 7:36 AM
Harbormaster failed remote builds in B19752: Diff 23973!
yuhan requested review of this revision.Oct 19 2020, 5:01 PM
yuhan retitled this revision from rfc EphemeralAddressStorage to RFC: DurableAddressStorage and EphemeralAddressStorage.Wed, Oct 28, 3:42 AM
yuhan edited the summary of this revision. (Show Details)
yuhan added reviewers: cdecarolis, sandyryza, schrockn.
python_modules/dagster/dagster/core/storage/asset_store.py
98