Enable memoization on every executor (#5506)


Enable memoization on every executor (#5506)

  • [Memoization 12/13] Fix bug with out-of-process memoization

Somehow missed this with tests, but since the execution plan is constructed from scratch in the step execution process for out of process executors, two issues occur:

  1. Doesn't pick up that version resolution has already occurred.
  2. Thinks that we're in the re-execution case, since step_keys_to_execute is not None.

The fix I went for carries over the step output versions from the orchestration process, and explicitly disables rebuilding the memoized plan within the step worker.

A different approach would be to construct the execution plan from a snapshot and then subset that one, but I'm not sure how to make that happen given that snapshots expect a pipeline snapshot as well.

Test Plan: Added a test to ensure multiprocess works with memoization. Also verified proper memoization behavior via Dagit. integration

Reviewers: sandyryza, alangenfeld

Differential Revision: https://dagster.phacility.com/D9220

thread instance through to external execution plan creation

  • Rebase into crag
  • test fix for test_error_dag_containerized
  • Addressed old comments from phab diff