Page MenuHomeElementl

Sketch of AssetOutput based lineage
Changes PlannedPublicDraft

Authored by owen on Feb 8 2021, 7:33 PM.

Details

Reviewers
None
Summary

I wanted to think of potential ways that we could allow users to specify AssetMaterializations in-line without having to restructure their code too much. There are still plenty of considerations to account for, and is heavily inspired by D6170.

The main difference between these two diffs is the idea that we can let the user specify storing/loading functions for the AssetKeys, rather than forcing them to write functions that will evaluate identically on either side of the input/output boundary. For users that are doing all their execution in the same process, the default IOManager can handle things in-memory behind the scenes, and for users that are not doing this, they already have to specify IOManagers, so the storing/loading functions here should not be too much extra work. We an also provide some useful defaults as optional mixins (things like file system asset key managers etc.)

Test Plan

This is just a demo, but the test I wrote is passing

Diff Detail

Repository
R1 dagster
Branch
asset_outputs
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 8 2021, 7:52 PM
Harbormaster failed remote builds in B25421: Diff 31009!

fixed some linting issues

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 8 2021, 8:09 PM
Harbormaster failed remote builds in B25428: Diff 31017!