Page MenuHomePhabricator

[3/n] Include pipeline snapshot and execution plan snapshot on system execution context
AbandonedPublic

Authored by dgibson on Jan 21 2021, 4:20 PM.

Details

Reviewers
alangenfeld
max
Summary

this is laying groundwork for a future where run worker / top-level pipeline execution has the option to just use these snapshots from the runs DB (and not load user code). Step execution would load the actual pipeline and solid definitions as before.

Test Plan

BK

Diff Detail

Repository
R1 dagster
Branch
rmpipelinedef3
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 21 2021, 4:54 PM
Harbormaster failed remote builds in B24635: Diff 29998!
dgibson retitled this revision from Include pipeline snapshot and execution plan snapshot on system execution context to [3/n] Include pipeline snapshot and execution plan snapshot on system execution context.Jan 22 2021, 3:36 PM

mostly just concerned about leaving stuff around we don't need - lets not add the execution plan snapshot if we dont need it

python_modules/dagster/dagster/core/execution/api.py
111–113

do you think we are on trajectory for ExecutionPlan to replace the need for the plan snapshot? If you don't use this snapshot in your stack lets drop it?

209

this call can be costly so we should probably ensure the snapshotting call is memoized

This revision is now accepted and ready to land.Jan 22 2021, 4:23 PM