cc @yuhan I feel like this is pretty related to the load-from-other-run-in-the-run-group look-ups we are doing inefficiently right now. Seems like there might be a way for this and that to work the same or be modeled the same

what is the right way to opt-in to memoization behavior? Is it a toggle on the Job? Something else? I don't know if I have a good answer.

In D8963#232989, @alangenfeld wrote:

what is the right way to opt-in to memoization behavior? Is it a toggle on the Job? Something else? I don't know if I have a good answer.

A toggle is what I have been thinking as well, but I'm also thinking about it from the dagit perspective, that it could be nice to have a checkbox like: memoize this run, or something. I think ideally it's a property of the job that it's whitelisted for memoization, but in a way that it doesn't always have to be run that way. I think some sort of toggle makes sense from that perspective.

cc @alangenfeld I
think it's also worth considering: what end state do we want to reach? Do we envision memoization to always be opt-in? Or could we at some point make it opt-out?
Making it opt-out would require making our default io managers memoizable (which should be possible in my eyes), and providing a default version on all solids/resources. Is that a crazy notion?

Do we envision memoization to always be opt-in? Or could we at some point make it opt-out?

I think we would have to have some pretty high confidence that the out of the box experience would work knowing 0 about the code happening in all the solids which seems like a reach to me. My intuition is that we should assume the worst case conditions happening in user code by default and have user action to sign off saying "ya my code meets these assumptions allowing this memoization scheme to work".

I could be convinced otherwise, but I think the path to doing so is to continue polishing the opt-in experience until the act of opting in becomes the worst part of the experience.

I agree with Alex regarding the opt-in question

the inline comments for the tags bit is still an open area of concern

in terms of modeling similar "load-from-other-run-in-the-run-group look-ups" in the same way, i could see a path forward that we consolidate logics into something like get_retry_steps_from_execution_plan - at a high level, all the paths are basically fn(historical execution plann, logs) -> execution_plan.

but i don't think it's a priority for this stack. filed an issue for tracking the potential refactor: https://github.com/dagster-io/dagster/issues/4418

cc @alangenfeld regarding tags:

• the latest rev should support switching on/off memoization from run via tag
• the versioned fs io manager doesn't actually support switching off memoization (always expects a version, version information only computed during memoization right now). Not to say people haven't created custom io managers that do support that.
• I think we ultimately want to switch away from using tags as a means of toggling memoization (https://dagster.phacility.com/D9061) provides an alternative way, but makes it more difficult to toggle off.

Which brings us back to Alex's initial q: what's the right way to toggle memoization on and off?

My thinking rn is that maybe it's best to surface it differently from different entrypoints, but more or less have it be a property of the run that we can point back to?

Which brings us back to Alex's initial q: what's the right way to toggle memoization on and off?

I'd like to switch it on/off in dagit, so making it toggleable via either config or tag sounds good to me - if we have it defined on jobs, imo it'd be less efficient for development like ml training.

In D8963#237378, @yuhan wrote:

Which brings us back to Alex's initial q: what's the right way to toggle memoization on and off?

I'd like to switch it on/off in dagit, so making it toggleable via either config or tag sounds good to me - if we have it defined on jobs, imo it'd be less efficient for development like ml training.

Maybe we can keep the underlying implementation as tags - and thus easily allow for switching off from basically every entrypoint - but foist the tag on when someone indicates that they want to use memoization by providing a memoization strategy or something. How bad would it be to implicitly add a tag to a run?

in terms of user experience, i think it's fine to implicitly add a tag to a run - some other features work like that, e.g. we implicitly add parent_run_id and root_run_id as tags to new runs, and solid/step selection too. besides, it's also good that users can view those info in the Runs page such as:

to clarify, i agree that tags are less robust as it doesn't have good checks and could get lost - that may be the trade-off we are making or config may be a better alternative bc it's schematized.

In D8963#237434, @yuhan wrote:

in terms of user experience, i think it's fine to implicitly add a tag to a run - some other features work like that, e.g. we implicitly add parent_run_id and root_run_id as tags to new runs, and solid/step selection too. besides, it's also good that users can view those info in the Runs page such as:

to clarify, i agree that tags are less robust as it doesn't have good checks and could get lost - that may be the trade-off we are making or config may be a better alternative bc it's schematized.

While the config approach is definitely a larger scale change - it would allow us to toggle with an actual boolean flag as opposed to string "true" and string`"false"` values, which make me so very sad.