Finishing this diff in the following PRs:

• IO for output notebook https://github.com/dagster-io/dagster/pull/4490
• handle output from notebook's yield_result calls https://github.com/dagster-io/dagster/pull/4496
• load input into notebooks https://github.com/dagster-io/dagster/pull/4500

my hunch is later in the stack (after D9085), this shouldn't be too large of a change, e.g. could potentially toggle is on io manager config?

in terms of user experience, i think it's fine to implicitly add a tag to a run - some other features work like that, e.g. we implicitly add parent_run_id and root_run_id as tags to new runs, and solid/step selection too. besides, it's also good that users can view those info in the Runs page such as:

Which brings us back to Alex's initial q: what's the right way to toggle memoization on and off?

trying enum

in terms of modeling similar "load-from-other-run-in-the-run-group look-ups" in the same way, i could see a path forward that we consolidate logics into something like get_retry_steps_from_execution_plan - at a high level, all the paths are basically fn(historical execution plann, logs) -> execution_plan.

gql

loosen handle_run_event error
add error handling to in mem storage

yup pipeline failure sensor is already nested under the run status sensor and solid hooks are still linked - i didn't update the failure sensor section in this diff.

DagsterRunNotFoundError

this is great!

PipelineSensorCursor -> RunStatusSensorCursor + backcompat fallback

one alternative would be to extend step_selection to also take an enum entry or marker type for describing a step selection to be calculated by dagster.

we could

• extend step_selection to take StepSelection.FROM_FAILURE), step_selection=StepSelection.from_selection(["some_solid*"]), and ["some_solid*"] (backcompat)
• or a more generic arg reexecute_option and deprecate step_selection, it'd take ReexecuteOption.FROM_FAILURE, ReexecuteOption.from_selection(["some_solid*"]))

not directly related to this diff: how do people feel about having a separate new-project cli for the crag stuff?

PipelineSensor -> RunStatusSensor

black

if it's version conflict, the make build cmd won't hard fail so BK won't be able to capture it. instead, if anything errors in the middle, it'd print out the errors and results empty json output for the ones that has conflict.
if api docs rendering is good, then it's good to go.

• run_status_sensor
• added test to cover the run interleave case for both run-sharded storage and non-run-sharded

can we make sure the make build in docs still works - iirc we had some lib version related issue when building the api docs. i could be wrong but just wanted to make sure unpinning it wont break the docs build.

For started events, it's expected that I fire the sensor even though by the time I've fired it, the run might have already changed state?

yes bc it's looking at the events not the pipeline statuses, which i think is the desired behavior. alternatively, if we look at the run status, the evaluation may miss firing if the state changes faster than the tick interval.

rebase

fix sitemap

disable docs next to unblock bk

Q management. lets do it after moving hacker news to public repo

yes plz

no context either.. but this looks good to me

rebase

up

defer to other reviewers as I'm not qualified enough to review the content here

This approach makes sense to me. But let's make sure this content doesn't conflict with the "recommended pattern" below https://docs.dagster.io/concepts/types#associating-dagster-types-with-python-types