- User Since
- Feb 28 2020, 11:37 PM (85 w, 6 d)
Fri, Oct 8
Sep 3 2021
Aug 16 2021
Aug 12 2021
Finishing this diff in the following PRs:
Aug 2 2021
my hunch is later in the stack (after D9085), this shouldn't be too large of a change, e.g. could potentially toggle is on io manager config?
Jul 30 2021
Which brings us back to Alex's initial q: what's the right way to toggle memoization on and off?
Jul 29 2021
in terms of modeling similar "load-from-other-run-in-the-run-group look-ups" in the same way, i could see a path forward that we consolidate logics into something like get_retry_steps_from_execution_plan - at a high level, all the paths are basically fn(historical execution plann, logs) -> execution_plan.
Jul 28 2021
Jul 27 2021
loosen handle_run_event error
add error handling to in mem storage
yup pipeline failure sensor is already nested under the run status sensor and solid hooks are still linked - i didn't update the failure sensor section in this diff.
Jul 26 2021
Jul 23 2021
this is great!
Jul 22 2021
PipelineSensorCursor -> RunStatusSensorCursor + backcompat fallback
Jul 21 2021
one alternative would be to extend step_selection to also take an enum entry or marker type for describing a step selection to be calculated by dagster.
- extend step_selection to take StepSelection.FROM_FAILURE), step_selection=StepSelection.from_selection(["some_solid*"]), and ["some_solid*"] (backcompat)
- or a more generic arg reexecute_option and deprecate step_selection, it'd take ReexecuteOption.FROM_FAILURE, ReexecuteOption.from_selection(["some_solid*"]))
Jul 20 2021
not directly related to this diff: how do people feel about having a separate new-project cli for the crag stuff?
PipelineSensor -> RunStatusSensor
Jul 19 2021
if it's version conflict, the make build cmd won't hard fail so BK won't be able to capture it. instead, if anything errors in the middle, it'd print out the errors and results empty json output for the ones that has conflict.
if api docs rendering is good, then it's good to go.
- added test to cover the run interleave case for both run-sharded storage and non-run-sharded
can we make sure the make build in docs still works - iirc we had some lib version related issue when building the api docs. i could be wrong but just wanted to make sure unpinning it wont break the docs build.
Jul 16 2021
Jul 15 2021
Jul 14 2021
For started events, it's expected that I fire the sensor even though by the time I've fired it, the run might have already changed state?
yes bc it's looking at the events not the pipeline statuses, which i think is the desired behavior. alternatively, if we look at the run status, the evaluation may miss firing if the state changes faster than the tick interval.
Jul 13 2021
Jul 9 2021
Jul 8 2021
Jul 7 2021
Q management. lets do it after moving hacker news to public repo
no context either.. but this looks good to me
Jul 6 2021
defer to other reviewers as I'm not qualified enough to review the content here
This approach makes sense to me. But let's make sure this content doesn't conflict with the "recommended pattern" below https://docs.dagster.io/concepts/types#associating-dagster-types-with-python-types