Enable memoization on every executor (#5506)
- [Memoization 12/13] Fix bug with out-of-process memoization
Somehow missed this with tests, but since the execution plan is constructed from scratch in the step execution process for out of process executors, two issues occur:
- Doesn't pick up that version resolution has already occurred.
- Thinks that we're in the re-execution case, since step_keys_to_execute is not None.
The fix I went for carries over the step output versions from the orchestration process, and explicitly disables rebuilding the memoized plan within the step worker.
A different approach would be to construct the execution plan from a snapshot and then subset that one, but I'm not sure how to make that happen given that snapshots expect a pipeline snapshot as well.
Test Plan: Added a test to ensure multiprocess works with memoization. Also verified proper memoization behavior via Dagit. integration
Reviewers: sandyryza, alangenfeld
Differential Revision: https://dagster.phacility.com/D9220
thread instance through to external execution plan creation
- Rebase into crag
- test fix for test_error_dag_containerized
- Addressed old comments from phab diff