implementation of [[ https://elementl.quip.com/TN6ZAFWkPkSb/Asset-Storage | Asset Storage ]]
This diff moves the dagster type loader and materializer invocation from `execute_step.py` to `intermediate_storage`. It also adds an asset cache in it, which is a mapping from asset.key to a list of assets and will be used to support versioning and memoization.
It creates a base for intermediate storage (which we plan to rename to asset storage) to
- handle step_input loading cases (from intermediate if it has intermediates, from config if value provided via config or loader set)
+ status-quo: [[ https://dagster.phacility.com/source/dagster/browse/master/python_modules/dagster/dagster/core/execution/plan/execute_step.py$442 | _input_values_from_intermediate_storage ]]
- handle step_output materializing cases (set intermediate or use materializer) and therefore merge the intermediate and asset materialization.
+ status-quo: [[ https://dagster.phacility.com/source/dagster/browse/master/python_modules/dagster/dagster/core/execution/plan/execute_step.py$379 | _create_output_materializations ]]
next steps:
1. merge the cases described above in intermediate storage
2. start working on the user-facing api design - how we can enable data to be passed by reference + reduce boiler plates for users to specify that option