Change Details

implementation of [[ https://elementl.quip.com/TN6ZAFWkPkSb/Asset-Storage | Asset Storage ]] This diff moves the dagster type loader and materializer invocation from `execute_step.py` to `intermediate_storage`. It also adds an asset cache in it, which is a mapping from asset.key to a list of assets and will be used to support versioning and memoization. It creates a base for intermediate storage (which we plan to rename to asset storage) to - handle step_input loading cases (from intermediate if it has intermediates, from config if value provided via config or loader set) + status-quo: [[ https://dagster.phacility.com/source/dagster/browse/master/python_modules/dagster/dagster/core/execution/plan/execute_step.py$442 | _input_values_from_intermediate_storage ]] - handle step_output materializing cases (set intermediate or use materializer) and therefore merge the intermediate and asset materialization. + status-quo: [[ https://dagster.phacility.com/source/dagster/browse/master/python_modules/dagster/dagster/core/execution/plan/execute_step.py$379 | _create_output_materializations ]] next steps: 1. merge the cases described above in intermediate storage 2. start working on the user-facing api design - how we can enable data to be passed by reference + reduce boiler plates for users to specify that option

implementation of [[ https://elementl.quip.com/TN6ZAFWkPkSb/Asset-Storage | Asset Storage ]] This diff moves the dagster type loader and materializer invocation from `execute_step.py` to `intermediate_storage`. It also adds an asset cache in it, which is a mapping from asset.key to a list of assets and will be used to support versioning and memoization. It creates a base for intermediate storage (which we plan to rename to asset storage) to - handle step_input loading cases (from intermediate if it has intermediates, from config if value provided via config or loader set) * status-quo: [[ https://dagster.phacility.com/source/dagster/browse/master/python_modules/dagster/dagster/core/execution/plan/execute_step.py$442 | _input_values_from_intermediate_storage ]] - handle step_output materializing cases (set intermediate or use materializer) and therefore merge the intermediate and asset materialization. * status-quo: [[ https://dagster.phacility.com/source/dagster/browse/master/python_modules/dagster/dagster/core/execution/plan/execute_step.py$379 | _create_output_materializations ]] next steps: 1. merge the cases described above in intermediate storage 2. start working on the user-facing api design - how we can enable data to be passed by reference + reduce boiler plates for users to specify that option

implementation of [[ https://elementl.quip.com/TN6ZAFWkPkSb/Asset-Storage | Asset Storage ]] This diff moves the dagster type loader and materializer invocation from `execute_step.py` to `intermediate_storage`. It also adds an asset cache in it, which is a mapping from asset.key to a list of assets and will be used to support versioning and memoization. It creates a base for intermediate storage (which we plan to rename to asset storage) to - handle step_input loading cases (from intermediate if it has intermediates, from config if value provided via config or loader set) +* status-quo: [[ https://dagster.phacility.com/source/dagster/browse/master/python_modules/dagster/dagster/core/execution/plan/execute_step.py$442 | _input_values_from_intermediate_storage ]] - handle step_output materializing cases (set intermediate or use materializer) and therefore merge the intermediate and asset materialization. +* status-quo: [[ https://dagster.phacility.com/source/dagster/browse/master/python_modules/dagster/dagster/core/execution/plan/execute_step.py$379 | _create_output_materializations ]] next steps: 1. merge the cases described above in intermediate storage 2. start working on the user-facing api design - how we can enable data to be passed by reference + reduce boiler plates for users to specify that option