Page MenuHomePhabricator

RFC: make AssetStores configurable per output
AbandonedPublic

Authored by sandyryza on Nov 24 2020, 4:29 PM.

Details

Summary

This enables asset stores to define a config schema that applies to each output. This enables us to support the output-related user stories described in https://elementl.quip.com/scopA6J0p5R3/InputOutput-Parameterization.

An assumption backing this direction is that there's a spectrum between "pure config-driven output storing" and "pure fixed location-driven output storing", and that users are likely to want to do things in the middle.

  • Pure config-driven output storing would be something like "store the output of solid X using the format and path specified at runtime by the analyst".
  • Pure fixed location-driven output storing would be something like "always store the output of solid X in table Y".
  • "Something in the middle" would be "store the output of solid X in a file with a name given at runtime by the analyst, using a directory and file format determined by the pipeline author".

This change aims at:

  • Making it more reasonable to support the third bullet.
  • Moving towards a world in which there's a single, user-definable code path for storing outputs, instead of type materializers vs. asset stores.
  • Moving towards the same for inputs.

Particularly wanted eyes on some of the controversial bits:

  • AssetStoreDefinitions that extend ResourceDefinitions.
  • Solid-level config that depends on what resource definitions are selected in the mode.

I don't like the name "store_outputs". Just a placeholder until we think of something better or consolidate with "outputs".

Take a look at test_asset_store_with_config for an example.

Test Plan

bk

Diff Detail

Repository
R1 dagster
Branch
input-scenarios (branched from master)
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Nov 24 2020, 4:46 PM
Harbormaster failed remote builds in B21658: Diff 26289!
sandyryza edited the summary of this revision. (Show Details)
sandyryza retitled this revision from make AssetStores configurable per output to RFC: make AssetStores configurable per output.Nov 25 2020, 2:40 AM

readd me to new diff once it is in a more reviewable state