Page MenuHomeElementl

asset-storage-2 move type loader/materializer into intermediate_storage and add cache

Authored by yuhan on Sep 22 2020, 6:52 PM.



implementation of Asset Storage

This diff moves the dagster type loader and materializer invocation from to intermediate_storage. It also adds an asset cache in it, which is a mapping from asset.key to a list of assets and will be used to support versioning and memoization.

It creates a base for intermediate storage (which we plan to rename to asset storage) to

  • handle step_output materializing cases (set intermediate or use materializer) and therefore merge the intermediate and asset materialization.

next steps:

  1. merge the cases described above in intermediate storage
  2. start working on the user-facing api design - how we can enable data to be passed by reference + reduce boiler plates for users to specify that option
Test Plan


Diff Detail

R1 dagster
Lint Passed
No Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 22 2020, 7:12 PM
Harbormaster failed remote builds in B18585: Diff 22564!

user_code_error_boundary and comments

These changes don't raise any particular red flags, but it's a little difficult for me to assess them without understanding the changes in user-facing experience that they're driving. Is that written out somewhere?