Jul 22 2021
@jordansanders how strongly do you feel about stars vs. dashes? I don't mind changing this time, but fairly annoying if that needs to be another step in the release guide. (the stars are generated by the Quip markdown export).
This looks great
Jul 21 2021
This looks great
I agree with Alex regarding the opt-in question
Jul 20 2021
abandoning this in favor of a version that will live inside dagster core
Do you have thoughts on what the dagit experience should be like? I.e. where does graph description vs. job description show up?
Jul 19 2021
Thoughts on replacing the duration with the most recent start time in the header? I think the most recent start time is a more useful piece of information than the duration.
Jul 16 2021
great that this already works
Jul 15 2021
Jul 14 2021
Thoughts on calling this pipeline_event_sensor? pipeline_sensor could be interpreted as a sensor that kicks off a pipeline.
I left a final comment that should be addressed before merging. Otherwise, looks great.
Jul 13 2021
Oops accepted by accident
I had originally been a strong proponent of this, but, having reviewed the resources in our dogfood pipelines, I'm less sure. It's only really the IOManagers that would benefit from this. I wonder if it would make more sense to just make it a little less verbose to make a hardcoded resource. Right now it requires ResourceDefinition.hardcoded_resource. We could make hardcoded_resource a top-level API?
Thanks a ton for picking this up. To make it accessible to users, it needs a few more things:
- Docstrings for the _partitioned_config functions.
- Experimental warnings.
- Inclusion in the experimental API docs page.
- Inclusion in the top-level public API.
Jul 12 2021
test failure is unrelated
Test failure is unrelated
Jul 8 2021
Jul 7 2021
I chatted with Alex offline and he convinced me on the approach here. I'll leave it to @cdecarolis for final review, but consider my concerns addressed.
This looks great!
Resigning because I don't have a ton of context.
I might not be the best person to review this, but looks reasonable to me.
Jul 6 2021
To clarify, you mean just fs_io_manager and not also the defaulting of multiprocess execution?
Jul 2 2021
This looks great.
Going to defer to Alex on this one
+1 for doing this. Resigning because I'm not qualified to comment on the implementation.
+1 to adding executor_def on jobs. The only part of this change that makes me nervous is switching the default to fs_io_manager. execute_in_process(**job.unpack_for_test()) feels gross to me. This could potentially give us the best of both worlds: https://dagster.phacility.com/D8695?