Page MenuHomeElementl

Load execution plan from the snapshot on the run when resume retrying
ClosedPublic

Authored by dgibson on Jul 23 2021, 4:05 PM.
Tags
None
Referenced Files
F2439856: D9041.diff
Tue, Aug 16, 9:57 PM
F2439410: D9041.id42224.diff
Tue, Aug 16, 7:40 PM
F2439409: D9041.id42204.diff
Tue, Aug 16, 7:40 PM
F2439408: D9041.id.diff
Tue, Aug 16, 7:40 PM
Unknown Object (File)
Sun, Aug 14, 4:49 AM
Unknown Object (File)
Fri, Aug 12, 2:58 PM
Unknown Object (File)
Fri, Aug 12, 2:57 PM
Unknown Object (File)
Tue, Aug 9, 5:38 AM
Subscribers
None

Details

Summary

Right now when you re-execute a pipeline from failure, we load an execution plan over gRPC not once, not twice, but thrice - once from the parent run to figure out the steps, once from the child run without any steps to validate that the steps are correct, and then a third time with the steps.

This diff takes the first call out of the equation by loading the execution plan directly from the snapshot.

Test Plan

BK, re-execute a pipeline from failure in dagit, verify it works and log to verify that the number of gRPC calls goes from 3 to 2

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 23 2021, 4:28 PM
Harbormaster failed remote builds in B34143: Diff 42204!

coolbeans

python_modules/dagster/dagster/core/execution/plan/resume_retry.py
46

nit: a dagster exception maybe

This revision is now accepted and ready to land.Jul 23 2021, 6:52 PM
This revision was landed with ongoing or failed builds.Jul 23 2021, 7:15 PM
This revision was automatically updated to reflect the committed changes.