- User Since
- Jul 20 2020, 3:42 PM (54 w, 15 h)
Get rid of code example
Rebase + allow tag to toggle memoization off
Sat, Jul 31
Fri, Jul 30
cc @alangenfeld regarding tags:
Account for run tags to determine whether to use memoization
Mypy execution plan snapshot, add a test to ensure that we don't carry around None values for step_output_versions
Thu, Jul 29
Reimplement using VersionStrategy instance, add code example
Implement VersionStrategy class
Wed, Jul 28
While I can def see the utility of the idea, this interface feels a bit off to me. From what I understand, the difference between nodes and logical assets isn't entirely cut and dry, and I'm wondering if people shouldn't be able to just add nodes to their list of assets, and we infer them as such / foist them into an asset?
Tue, Jul 27
cc @yuhan @alangenfeld might abandon this in favor of https://dagster.phacility.com/D9085, but the discussion is still relevant I think. I'm certainly open to having a versioned_outputs directory or something at that layer.
This doesn't actually work
@alangenfeld upon further examination, I don't think the execute_plan solution can work. Even if we keep around the original executors, we still need to create a new pipeline/mode in order to switch out the io manager. Then, we'll get a different set of complaints regarding the fact that we're using a non-persistent IO manager with an out of proc executor. I think we're forced to either do the config replacement or the permissive thing.
cc @alangenfeld ^ bc I forgot to tag you
have the "special" in process executor change make the whole execution section permissive at RunConfigSchema generation time - I think this is promising
I know this isn't a huge regression, but it still feels like a regression to me. I feel like it's a pretty core change for what is fundementally a workaround.
Attempting new strategy - ingest solid definition and produce version from it.
Mon, Jul 26
Respin with approach Sandy & I discussed: A function that takes in a solid definition and produces a version from it.
Finish updating snapshots
Fix snapshots and cli tests
Fix snapshots (tests are passing on local)
Fix issue with version retrieval
Fri, Jul 23
Revamp tests, revamp serialization of step output versions, more efficient resource implementation
Thu, Jul 22
Add description to dev loop tests
LGTM aside from testing nit
Get rid of universal step_keys_to_execute check
Address comments: make implementations more readable, add documentation, add hashing tests, fix errors
Wed, Jul 21
Fix issue with config post-processing
Give implementations another pass. Finish addressing comments
Update implementation and addressed most comments
Tue, Jul 20
cc @alangenfeld I
think it's also worth considering: what end state do we want to reach? Do we envision memoization to always be opt-in? Or could we at some point make it opt-out?
Making it opt-out would require making our default io managers memoizable (which should be possible in my eyes), and providing a default version on all solids/resources. Is that a crazy notion?
Still more tests to fix but looking for feedback on the general idea
Fix docs example tests
Update yet more snapshot tests
Fix more snap tests
I think this looks good. Some of the checks I don't have full context on but the lack of mypy complaining seems like it should be alright (until we mypy more of the system and find out something else is broken).
Mon, Jul 19
Fri, Jul 16
Add graph container when constructing solid
@alangenfeld how would you feel about requiring both solid and container to use the context.solid param? That way we could avoid a lot of the nastiness that results from making the container optional on Solid
Thu, Jul 15
Nice. Maybe move it a bit higher on the list?
Implement an approach that preserves correctness - at great great cost
Wed, Jul 14
Gonna go with separate args for minute and hour to keep in theme.