Page MenuHomeElementl

Load execution plan from the snapshot on the run when resume retrying
ClosedPublic

Authored by dgibson on Jul 23 2021, 4:05 PM.
Tags
None
Referenced Files
Unknown Object (File)
Fri, May 12, 12:40 PM
Unknown Object (File)
Wed, May 10, 4:09 AM
Unknown Object (File)
Tue, May 9, 4:49 PM
Unknown Object (File)
Mon, May 8, 1:27 PM
Unknown Object (File)
Apr 21 2023, 4:31 AM
Unknown Object (File)
Apr 15 2023, 6:41 PM
Unknown Object (File)
Mar 23 2023, 7:26 PM
Unknown Object (File)
Mar 17 2023, 2:30 PM
Subscribers
None

Details

Summary

Right now when you re-execute a pipeline from failure, we load an execution plan over gRPC not once, not twice, but thrice - once from the parent run to figure out the steps, once from the child run without any steps to validate that the steps are correct, and then a third time with the steps.

This diff takes the first call out of the equation by loading the execution plan directly from the snapshot.

Test Plan

BK, re-execute a pipeline from failure in dagit, verify it works and log to verify that the number of gRPC calls goes from 3 to 2

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 23 2021, 4:28 PM
Harbormaster failed remote builds in B34143: Diff 42204!

coolbeans

python_modules/dagster/dagster/core/execution/plan/resume_retry.py
46

nit: a dagster exception maybe

This revision is now accepted and ready to land.Jul 23 2021, 6:52 PM
This revision was landed with ongoing or failed builds.Jul 23 2021, 7:15 PM
This revision was automatically updated to reflect the committed changes.