HomeElementl

[memoization 3/n] move core memoization logic to live on the execution plan

Description

[memoization 3/n] move core memoization logic to live on the execution plan

Summary:
This diff moves the core memoization logic to live on the execution plan. This, among other changes, represent a major usability improvement to the memoization system.

  • Makes memoization available across all modes of execution. Dagit, graphql, cli, etc.
  • Improved error messages when code versions are not set, that will hopefully point users in the right direction instead of silently failing.
  • Only initializes the resources used for the io manager that is being initialized. Previously, all resources on the mode would be initialized.
  • Storing version information on the plan means that it only needs to be computed once, even across process boundaries. This represents a major improvement to the naive way it was computed before.

Test Plan: Modified unit tests to take advantage of this new approach, and I have a checklist of more tests I need to add.

Reviewers: sandyryza, alangenfeld, yuhan

Reviewed By: yuhan

Subscribers: yuhan

Differential Revision: https://dagster.phacility.com/D8963