All the info we need are:
- a write/read method
- a path where the object will be loaded from or materialized to
- the object
1) write/read method D4697
write
- object_store.set_object(path, obj, serialization_strategy)
- dagster_type.materializer.materialize_runtime_values(context, config_value, obj)
read
- object_store.get_object(path, serialization_strategy)
- dagster_type.loader.construct_from_config_value(context, config_value)
so the processing priority could be:
- use dagster_type.materializer/loader to handle intermediates if it's address.config_value is provided
- else, use object_store and path to handle intermediates
2) a path where the object will be loaded from or materialized to D4579
Address: either path or config_value
- path: default or used for object_store
- this is the default setting if the user doesn't specify anything
- get the path from Output(value, Address(path, object_store)) where the user can configure storage per output via object_store + the path to get/set
- config_value: used for dagster type loader/materializer
- get the config_value from solid_config.outputs. see [1]
- get the config_value from Output(value, address=Address(config_value)) in a solid. this would overwrite [1]. see [2]
Appendix
https://excalidraw.com/#json=5199594807885824,GIXYFPn9KCzKryJwCLSwxA
current state:
proposal:
