@alangenfeld good point - I updated to give the graphs inputs and outputs
Thanks a ton for putting this together @dgibson!
Tue, Aug 3
I like it
Mon, Aug 2
Sat, Jul 31
Fri, Jul 30
I think this makes sense. I feel like there might be an edge case out there where this causes trouble, but having trouble thinking of it. With this, are we able to get rid of VersionedPickledObjectFilesystemIOManager?
I'm going to defer to others on this
Thu, Jul 29
I thought about it a bit more, and it seems like we already have a concept for "thing that should get run after a node" with SolidHooks. Certainly they are a bit more limited in functionality than Ops (hard to chain together), but considering they have access to the output object of the node they're applied to, as well as the resource system, it seems like they should be expressive enough for any alerting-type behavior, have the benefit of easy reusability, have the correct rerun behavior, and don't require the addition of a new concept. Curious to hear your thoughts.
I added docstrings and beefed up the tests - I think this is ready to go in for others to build on top of. It'll be hidden behind dagster.assets, so still not part of the public API.
Wed, Jul 28
I'm wondering if people shouldn't be able to just add nodes to their list of assets, and we infer them as such / foist them into an asset?
+1 to having a Strategy class that can version both solids and resources. IMO preferable to call it VersionStrategy or VersioningStrategy, because it's supplying versions, which may or may not be used for memoization.
Tue, Jul 27
@owen - I updated the diff to include io_manager_keys
Hmm maybe I misread and the pipeline failure sensor was already nested under the run status sensor? Accepting!
Mon, Jul 26
@alangenfeld One interesting problem that we've discussed in past iterations of this direction is how different @assets may materialize their input assets in to different in-memory types, controlled separately from how they are persisted. Thoughts on that? Something for a follow up?
Since pipeline failure sensors are the most common case, would it make sense to put them above the run status sensor? Alternatively, would it make sense to nest them inside?