Page MenuHomePhabricator

asset-storage-2 move type loader/materializer into intermediate_storage and add cache
AbandonedPublic

Authored by yuhan on Sep 22 2020, 6:52 PM.

Details

Reviewers
None
Summary

implementation of Asset Storage

This diff moves the dagster type loader and materializer invocation from execute_step.py to intermediate_storage. It also adds an asset cache in it, which is a mapping from asset.key to a list of assets and will be used to support versioning and memoization.

It creates a base for intermediate storage (which we plan to rename to asset storage) to

  • handle step_output materializing cases (set intermediate or use materializer) and therefore merge the intermediate and asset materialization.

next steps:

  1. merge the cases described above in intermediate storage
  2. start working on the user-facing api design - how we can enable data to be passed by reference + reduce boiler plates for users to specify that option
Test Plan

bk

Diff Detail

Repository
R1 dagster
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 22 2020, 7:12 PM
Harbormaster failed remote builds in B18585: Diff 22564!

user_code_error_boundary and comments

These changes don't raise any particular red flags, but it's a little difficult for me to assess them without understanding the changes in user-facing experience that they're driving. Is that written out somewhere?