Jul 29 2021
I thought about it a bit more, and it seems like we already have a concept for "thing that should get run after a node" with SolidHooks. Certainly they are a bit more limited in functionality than Ops (hard to chain together), but considering they have access to the output object of the node they're applied to, as well as the resource system, it seems like they should be expressive enough for any alerting-type behavior, have the benefit of easy reusability, have the correct rerun behavior, and don't require the addition of a new concept. Curious to hear your thoughts.
I added docstrings and beefed up the tests - I think this is ready to go in for others to build on top of. It'll be hidden behind dagster.assets, so still not part of the public API.
Jul 28 2021
I'm wondering if people shouldn't be able to just add nodes to their list of assets, and we infer them as such / foist them into an asset?
+1 to having a Strategy class that can version both solids and resources. IMO preferable to call it VersionStrategy or VersioningStrategy, because it's supplying versions, which may or may not be used for memoization.
Jul 27 2021
@owen - I updated the diff to include io_manager_keys
Hmm maybe I misread and the pipeline failure sensor was already nested under the run status sensor? Accepting!
Jul 26 2021
One interesting problem that we've discussed in past iterations of this direction is how different @assets may materialize their input assets in to different in-memory types, controlled separately from how they are persisted. Thoughts on that? Something for a follow up?
Since pipeline failure sensors are the most common case, would it make sense to put them above the run status sensor? Alternatively, would it make sense to nest them inside?
Jul 22 2021
@jordansanders how strongly do you feel about stars vs. dashes? I don't mind changing this time, but fairly annoying if that needs to be another step in the release guide. (the stars are generated by the Quip markdown export).
This looks great
Jul 21 2021
This looks great
I agree with Alex regarding the opt-in question
Jul 20 2021
abandoning this in favor of a version that will live inside dagster core
Do you have thoughts on what the dagit experience should be like? I.e. where does graph description vs. job description show up?
Jul 19 2021
Thoughts on replacing the duration with the most recent start time in the header? I think the most recent start time is a more useful piece of information than the duration.
Jul 16 2021
great that this already works
Jul 15 2021
Jul 14 2021
Thoughts on calling this pipeline_event_sensor? pipeline_sensor could be interpreted as a sensor that kicks off a pipeline.
I left a final comment that should be addressed before merging. Otherwise, looks great.
Jul 13 2021
Oops accepted by accident
I had originally been a strong proponent of this, but, having reviewed the resources in our dogfood pipelines, I'm less sure. It's only really the IOManagers that would benefit from this. I wonder if it would make more sense to just make it a little less verbose to make a hardcoded resource. Right now it requires ResourceDefinition.hardcoded_resource. We could make hardcoded_resource a top-level API?
Thanks a ton for picking this up. To make it accessible to users, it needs a few more things:
- Docstrings for the _partitioned_config functions.
- Experimental warnings.
- Inclusion in the experimental API docs page.
- Inclusion in the top-level public API.