Page MenuHomePhabricator

RFC: make AssetStores configurable per output

Authored by sandyryza on Nov 24 2020, 4:29 PM.



This enables asset stores to define a config schema that applies to each output. This enables us to support the output-related user stories described in

An assumption backing this direction is that there's a spectrum between "pure config-driven output storing" and "pure fixed location-driven output storing", and that users are likely to want to do things in the middle.

  • Pure config-driven output storing would be something like "store the output of solid X using the format and path specified at runtime by the analyst".
  • Pure fixed location-driven output storing would be something like "always store the output of solid X in table Y".
  • "Something in the middle" would be "store the output of solid X in a file with a name given at runtime by the analyst, using a directory and file format determined by the pipeline author".

This change aims at:

  • Making it more reasonable to support the third bullet.
  • Moving towards a world in which there's a single, user-definable code path for storing outputs, instead of type materializers vs. asset stores.
  • Moving towards the same for inputs.

Particularly wanted eyes on some of the controversial bits:

  • AssetStoreDefinitions that extend ResourceDefinitions.
  • Solid-level config that depends on what resource definitions are selected in the mode.

I don't like the name "store_outputs". Just a placeholder until we think of something better or consolidate with "outputs".

Take a look at test_asset_store_with_config for an example.

Test Plan


Diff Detail

R1 dagster
input-scenarios (branched from master)
Lint OK
No Unit Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Nov 24 2020, 4:46 PM
Harbormaster failed remote builds in B21658: Diff 26289!
sandyryza edited the summary of this revision. (Show Details)
sandyryza retitled this revision from make AssetStores configurable per output to RFC: make AssetStores configurable per output.Nov 25 2020, 2:40 AM

readd me to new diff once it is in a more reviewable state