• I'm skeptical that even supporting path in this is a good idea. I would rather see two asset stores: one that requires paths and one that automatically assigns for you. I think the in between is goofy.

sounds good. will update it.

arbitrary asset_metadata

path not required in default asset store

rebase /0

good job! this is looking a lot better

would be good to show how users can specify the solid config value in [1] and [2] too as you did in the test

flaky k8s?

can you screenshot the final view of the page for reviewing?

make AddressStorage configurale via instance yaml + docstring

@sandyryza

If set_asset is able to determine the address just from the context and the asset_metadata, can get_asset do that as well? I.e. do we need to make a call to a remote service, vs. deriving it from the data we already have available?

did you mean get_asset in AssetStore? what do you mean by a remote service? not sure i fully understand the question 😅

up

make AddressStore a new concept on instance -- completely decouple it from intermediate storage

separate from intermediate storage

up

We get to avoid adding and maintaining an "AddressStore" component that pipes addresses between steps.

in the working prototype of this api D4579, i made address_store a mapping inside IntermediateStorage. i think we can avoid making it an extra layer between intermediates and steps.

AssetAddress, AssetStore, intermediate_storage.address_store

AssetStore, AddressStore, AssetAddress

up

D4579

D4697

are we going to update all the outdated screenshots? can we do a pass at once to update them all together?
or is it something @max 's automated screenshot script can help?

py2

split diffs

up

Address

update call site in tests (airline_demo and dagster_pandas)

this looks good!

py2 syntax

• cross-run intermediates based on events -> reexecution works
• event - SET_EXTERNAL_OBJECT, GET_EXTERNAL_OBJECT
• example - pass address from Output inside a solid

limitation - dagster type loader and materializer has to be paired if they are used for intermediate operations

simplify examples - use solid_config.outputs to handle intermediates

simplify examples - use solid_config.outputs to handle intermediates

I kinda feel it's less robust to surface the variable by the literal arg name "context" and it could be consistent to all other entities (e.g. solid, resource, hook, etc) if the context is always the first positional arg.
my general sense was it's better to be restrict first and then lose the constraint than the opposite. I don't feel strongly about it in this case tho -- i trust your call

great start!

should we be more restrict to limit context to be a required arg and at the first position like all other decorators do?

experimental_arg_warning

experimental + DagsterInvalidAddressForAsset -> DagsterAddressIOError

this looks good to me.