Page MenuHomeElementl

RFC pipeline-level alerting - event based
Changes PlannedPublic

Authored by yuhan on Feb 12 2021, 8:36 AM.

Details

Reviewers
johann
Summary

this diff prototypes an event-based pipeline hook. this is really similar to the solid hooks i wrote before.

  • diffs related to solid hooks: D3977 and its related stack

imo the pieces are

  1. how to define a pipeline hook, e.g. alert on what
  2. how to specify a pipeline hook, e.g. triggered by which pipeline
  3. how to thread it so it works properly with all the dagster machinery, e.g. where it gets triggered, which layer we plug it into

this is really really hacky. it is working (see examples/hooks/pipeline_hooks.py) but i don't feel good about any of the piece here...

relevant discussion: https://github.com/dagster-io/dagster/issues/3613
i think it makes sense to thread the hook into the execution plan layer - it's on pipeline def now, which i dont think is good

Test Plan

python examples/hooks/pipeline_hooks.py

Screen Shot 2021-02-12 at 12.01.17 AM.png (554×2 px, 321 KB)

Diff Detail

Repository
R1 dagster
Branch
yuhan/pipeline-hook
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

yuhan edited the test plan for this revision. (Show Details)
yuhan published this revision for review.Feb 12 2021, 8:42 AM

i know this implementation is bad but it could give you something to start with before i get up :)

johann added a subscriber: johann.