See comment

yay

Get rid of code example

Rebase + allow tag to toggle memoization off

In D8963#237434, @yuhan wrote:

in terms of user experience, i think it's fine to implicitly add a tag to a run - some other features work like that, e.g. we implicitly add parent_run_id and root_run_id as tags to new runs, and solid/step selection too. besides, it's also good that users can view those info in the Runs page such as:

to clarify, i agree that tags are less robust as it doesn't have good checks and could get lost - that may be the trade-off we are making or config may be a better alternative bc it's schematized.

Fix plumbing

In D8963#237378, @yuhan wrote:

Which brings us back to Alex's initial q: what's the right way to toggle memoization on and off?

I'd like to switch it on/off in dagit, so making it toggleable via either config or tag sounds good to me - if we have it defined on jobs, imo it'd be less efficient for development like ml training.

cc @alangenfeld regarding tags:

Account for run tags to determine whether to use memoization

Up

Mypy execution plan snapshot, add a test to ensure that we don't carry around None values for step_output_versions

Reimplement using VersionStrategy instance, add code example

Implement VersionStrategy class

Up

Address comments

In D9061#236325, @alangenfeld wrote:

This strategy only really works for solids. A follow up diff will address the resources situation. Not sure if the best solution there is to keep the version argument on the resource decorator and make it not required to use memoization, or to add a resource_versioning_strategy argument (also not required) as well.

Hmm, I feel like I need to know the whole end state to evaluate this.

I think the f(solid) -> version pattern is reasonable, but should it be passed in just as a free function or something like a class with staticmethods that meets some interface MemoizationStrategy or something

While I can def see the utility of the idea, this interface feels a bit off to me. From what I understand, the difference between nodes and logical assets isn't entirely cut and dry, and I'm wondering if people shouldn't be able to just add nodes to their list of assets, and we infer them as such / foist them into an asset?

Nice

cc @yuhan @alangenfeld might abandon this in favor of https://dagster.phacility.com/D9085, but the discussion is still relevant I think. I'm certainly open to having a versioned_outputs directory or something at that layer.

This doesn't actually work

@alangenfeld upon further examination, I don't think the execute_plan solution can work. Even if we keep around the original executors, we still need to create a new pipeline/mode in order to switch out the io manager. Then, we'll get a different set of complaints regarding the fact that we're using a non-persistent IO manager with an out of proc executor. I think we're forced to either do the config replacement or the permissive thing.

Fix tests

cc @alangenfeld ^ bc I forgot to tag you

have the "special" in process executor change make the whole execution section permissive at RunConfigSchema generation time - I think this is promising

I know this isn't a huge regression, but it still feels like a regression to me. I feel like it's a pretty core change for what is fundementally a workaround.

Attempting new strategy - ingest solid definition and produce version from it.

Respin with approach Sandy & I discussed: A function that takes in a solid definition and produces a version from it.

Finish updating snapshots

Fix snapshots and cli tests

Fix snapshots (tests are passing on local)

Fix issue with version retrieval

Address comments.

Up

Revamp tests, revamp serialization of step output versions, more efficient resource implementation

Add description to dev loop tests

LGTM aside from testing nit

Get rid of universal step_keys_to_execute check

Address comments: make implementations more readable, add documentation, add hashing tests, fix errors

Fix issue with config post-processing

Snapshot up

Give implementations another pass. Finish addressing comments

Update implementation and addressed most comments

Second pass

cc @alangenfeld I
think it's also worth considering: what end state do we want to reach? Do we envision memoization to always be opt-in? Or could we at some point make it opt-out?
Making it opt-out would require making our default io managers memoizable (which should be possible in my eyes), and providing a default version on all solids/resources. Is that a crazy notion?

In D8963#232989, @alangenfeld wrote:

what is the right way to opt-in to memoization behavior? Is it a toggle on the Job? Something else? I don't know if I have a good answer.

In D8923#232886, @alangenfeld wrote:

feels like there should be a few more tests but im struggling to think of what cases make sense to test. reexecute_pipeline with explicit empty step selection?

Fix tests

Still more tests to fix but looking for feedback on the general idea

Fix docs example tests

Update yet more snapshot tests

Fix more snap tests

I think this looks good. Some of the checks I don't have full context on but the lack of mypy complaining seems like it should be alright (until we mypy more of the system and find out something else is broken).

Update snaps

Address comments

In D8550#232425, @alangenfeld wrote:

If we want to go back to the previous direction, we could walk / refer to the pipeline [1] structure when transforming the config.

The benefit this current direction has is providing something that we could be sent over graphql to dagit to allow the frontend to more gracefully handle these field aliases.

for what its worth I think this direction could get to a better feeling place with some iteration

Up

Add graph container when constructing solid

launch API attempt up next

@alangenfeld how would you feel about requiring both solid and container to use the context.solid param? That way we could avoid a lot of the nastiness that results from making the container optional on Solid

Nice. Maybe move it a bit higher on the list?

Implement an approach that preserves correctness - at great great cost

Gonna go with separate args for minute and hour to keep in theme.