Page MenuHomeElementl

Load execution plan from the snapshot on the run when resume retrying
ClosedPublic

Authored by dgibson on Jul 23 2021, 4:05 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sat, Jan 28, 5:28 PM
Unknown Object (File)
Tue, Jan 24, 5:51 AM
Unknown Object (File)
Mon, Jan 23, 7:17 PM
Unknown Object (File)
Jan 2 2023, 6:02 AM
Unknown Object (File)
Dec 16 2022, 1:56 PM
Unknown Object (File)
Nov 30 2022, 11:32 AM
Unknown Object (File)
Nov 18 2022, 1:37 AM
Unknown Object (File)
Nov 18 2022, 1:37 AM
Subscribers
None

Details

Summary

Right now when you re-execute a pipeline from failure, we load an execution plan over gRPC not once, not twice, but thrice - once from the parent run to figure out the steps, once from the child run without any steps to validate that the steps are correct, and then a third time with the steps.

This diff takes the first call out of the equation by loading the execution plan directly from the snapshot.

Test Plan

BK, re-execute a pipeline from failure in dagit, verify it works and log to verify that the number of gRPC calls goes from 3 to 2

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 23 2021, 4:28 PM
Harbormaster failed remote builds in B34143: Diff 42204!

coolbeans

python_modules/dagster/dagster/core/execution/plan/resume_retry.py
46

nit: a dagster exception maybe

This revision is now accepted and ready to land.Jul 23 2021, 6:52 PM
This revision was landed with ongoing or failed builds.Jul 23 2021, 7:15 PM
This revision was automatically updated to reflect the committed changes.