naming spitballing:

• pipeline_failure_sensor / pipeline_sensor which means it watches pipelines - accurate, no new noun
• pipeline_failure_monitor / pipeline_monitor which creates a new noun "monitor" but the name "monitor" is more intuitive as it describes the use case not the machinery

as im writing, i think im leaning towards pipeline_failure_sensor / pipeline_sensor so we wont be introducing new nouns too early

If a monitor ultimately boils down to a sensor, is it worth introducing a new noun? I.e. should we just call this pipeline_failure_sensor?

i was thinking of subclassing the sensor def to a PipelineSensorDefinition later in a diff, and we can have a @pipeline_sensor to address a generic sensing pipeline (i.e. monitoring) case.
it'd take arbitrary dagster event types and a set of pipeline pointer (e.g. pipeline_name), react to events and yield MonitorRequest. something like:

@pipeline_sensor(
   dagster_event_types=[DagsterEventType.PIPELINE_START],
   pipeline_names=["my_pipeline"], # pipeline_names=None means it watches all pipelines
) 
def alert_on_pipeline_start:
   ...

We probably will want to add a version that monitors / senses failures for a particular pipeline or set of pipelines. How do we imagine the name of that fitting in with the name of this API?

@pipeline_failure_sensor(pipeline_names=["my_pipeline"])
def alert_on_pipeline_failure_whitelisted:
   ...

This is looking pretty good to me. Not sure about the naming, but don't have great alternatives (pipeline_failure_watcher? watch_pipeline_failure?).

As an aside, wondering if the user function could optionally return some metadata that gets reported back in the engine event.

What about @pipeline_failure_sensor? @alangenfeld might also have opinions.

I'm ambivalent about pipeline_failure_sensor. I think it's nice that it exposes the underlying implementation, but not sure if we want to lock in the expectation that we support the same signature for the user function as you would a regular sensor evaluation function.

I'm ambivalent about pipeline_failure_sensor. I think it's nice that it exposes the underlying implementation, but not sure if we want to lock in the expectation that we support the same signature for the user function as you would a regular sensor evaluation function.

Yeah, I suppose it partly depends on how closely we want to connect the two concepts. Are there two separate user-facing concepts that share an underlying implementation, or is there a single flexible concept? When someone goes to view it in Dagit, should it show up in the list of sensors, or in a separate list of "monitors"? If someone wants to understand the object that's returned by the pipeline_failure_x decorator, do we direct them to the SensorDefinition API doc, or do we direct them to API doc for MonitorDefinition? If we later want to enable someone to define something that launches retries on pipeline failure, would we want to add a pipeline(s) param to the pipeline_failure_x, or would we want to expose a separate decorator and type?

bumping my early comment in case you missed it.

In D7476#206864, @yuhan wrote:

naming spitballing:

• pipeline_failure_sensor / pipeline_sensor which means it watches pipelines - accurate, no new noun
• pipeline_failure_monitor / pipeline_monitor which creates a new noun "monitor" but the name "monitor" is more intuitive as it describes the use case not the machinery

as im writing, i think im leaning towards pipeline_failure_sensor / pipeline_sensor so we wont be introducing new nouns too early

---

If a monitor ultimately boils down to a sensor, is it worth introducing a new noun? I.e. should we just call this pipeline_failure_sensor?

i was thinking of subclassing the sensor def to a PipelineSensorDefinition later in a diff, and we can have a @pipeline_sensor to address a generic sensing pipeline (i.e. monitoring) case.
it'd take arbitrary dagster event types and a set of pipeline pointer (e.g. pipeline_name), react to events and yield MonitorRequest. something like:

@pipeline_sensor(
   dagster_event_types=[DagsterEventType.PIPELINE_START],
   pipeline_names=["my_pipeline"], # pipeline_names=None means it watches all pipelines
) 
def alert_on_pipeline_start:
   ...

We probably will want to add a version that monitors / senses failures for a particular pipeline or set of pipelines. How do we imagine the name of that fitting in with the name of this API?

@pipeline_failure_sensor(pipeline_names=["my_pipeline"])
def alert_on_pipeline_failure_whitelisted:
   ...

i share all the concerns you two raised. i'm leaning towards pipeline_failure_sensor because

• it accurately describes what it is without introducing new nouns.
• one con of this is that it exposes the underlying implementation rather connecting to the use cases directly. but imo it's too early to decide a new noun as there are a few open questions that affect the decision.

here are a few next steps that will help us understand whether we need a new noun and what it would be called (sequenced by priority):

user facing wise:

• in dagit, how would a "pipeline failure sensor" be represented? currently in mvp, it makes sense to share the sensor page ui. going forward, if we think we need to lift it up to be a new job type (for example, we would have 3 separate pages sensor, schedule, and monitor in dagit).
• in python api, what a more generic pipeline sensor would look like? see the proposal above
• in python api, do we return a sensor def or something else like a subclass like MonitorDefinition? - i think the name of the returned type should be consistent with the decorator name, i.e. @pipeline_failure_sensor returns SensorDefinition or @pipeline_failure_monitor returns MonitorDefinition.

internal representation wise, would this underlying implementation eventually diverge from SensorDefinition and be another definition that's at the same level of sensor/schedule?

im proposing naming it pipeline_failure_sensor for now without exporting it, prototyping the above steps and see if we need a new noun

im proposing naming it pipeline_failure_sensor for now without exporting it, prototyping the above steps and see if we need a new noun

That sounds like a good direction to me. Thanks for your detailed response. I hadn't seen that comment you linked - sounds like you had thought through a bunch of this stuff already.