HomePhabricator

Remove reexecution_config, add memoization strategy for reexecution

Authored by prha on Tue, Nov 5, 8:04 AM.

Description

Remove reexecution_config, add memoization strategy for reexecution

Summary:
This diff restructures the reexecution flow to use previous_run_id and the existing step_key arguments on ExecutionParams instead of a separate ReexecutionConfig. It moves the step_keys_to_execute onto ExecutionPlan to thread through the APIs and engine. Adds a RetryMemoization class that handles the intermediate management across runs.

This contains a lot of the major refactor changes in D1294 (minus the resume/retry), but builds upon the RunConfig => PipelineRun change from D1329.

Test Plan: bk

Reviewers: #ft, schrockn, alangenfeld, max

Reviewed By: #ft, schrockn, alangenfeld

Subscribers: max, alangenfeld, natekupp, schrockn

Differential Revision: https://dagster.phacility.com/D1336

Details

Committed
prhaWed, Nov 6, 9:44 PM
Reviewer
Restricted Project
Differential Revision
D1336: Remove reexecution_config, add memoization strategy for reexecution
Parents
R1:6d0c3deb2c8a: pin ecdsa (moto dep), which has import resolving issues
Branches
Unknown
Tags
Unknown