Page MenuHomeElementl

email pipeline_failure_sensor
ClosedPublic

Authored by yuhan on Jun 24 2021, 8:17 PM.

Details

Summary

Similar to D8510, this diff provides out-of-box support for sending emails on pipeline failures over the SMTP

This is thin layer that doesn't address advanced (more secure) email auth connection.
So if an OSS user wants a more secure mechanism we could direct them to write their own.

from dagster import repository
from dagster.utils.email import make_email_on_pipeline_failure_sensor

email_on_pipeline_failure = make_email_on_pipeline_failure_sensor(
    email_from=os.getenv("ALERT_EMAIL_ADDRESS"),
    email_password=os.getenv("ALERT_EMAIL_PASSWORD"),
    email_to=["yuhan@elementl.com"],
    dagit_base_url="http://localhost",
)

@repository
def my_repo():
    return [my_pipeline, email_on_pipeline_failure]

api doc:

image.png (2×1 px, 400 KB)

One thing is open for discussion is how we get the email creds like sender email password.
This diff asks users to put sender email and password directly.
Alternatives are:

  1. ask for env vars instead
  2. to have an entry in the Dagster instance yaml to get those types of configs/tokens.
Test Plan

toy sensor and got emails

Screen Shot 2021-06-24 at 12.56.32 PM.png (784×1 px, 153 KB)

Diff Detail

Repository
R1 dagster
Branch
yuhan/slack-email-on-failure
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 24 2021, 8:55 PM
Harbormaster failed remote builds in B32603: Diff 40168!
Harbormaster returned this revision to the author for changes because remote builds failed.Jun 24 2021, 9:41 PM
Harbormaster failed remote builds in B32611: Diff 40176!

require email_from and email_password directly

yuhan published this revision for review.Jun 24 2021, 11:24 PM
yuhan added inline comments.
python_modules/dagster/dagster/utils/email.py
32–33

this is a workaround so gmail won't trim the content https://stackoverflow.com/questions/11078264/how-to-get-rid-of-show-trimmed-content-in-gmail-html-emails

w/o this line

image.png (1×1 px, 143 KB)

w/ this line:

image.png (1×1 px, 164 KB)

Would we be able to support something like a owner_emails pipeline tag so that people can get alerted about their pipelines?

Also, thoughts on including the name of the pipeline in the email subject? I'm imagining going through my email and only wanting to click into emails about pipelines that I'm interested in at that time. Maybe there are some flaky pipelines and I happen to know that failures there aren't worth investigating.

Would we be able to support something like a owner_emails pipeline tag so that people can get alerted about their pipelines?

I think it's doable through customizing pipeline_failure_sensor and we could later make it a built-in feature if we have a well thought out plan for some user/owner system in OSS.
Currently, users can attach tags to their pipelines and pipeline failure sensors have the access to pipeline tags via context.pipeline_run, so it's totally doable to encode that "pipeline owner" logic in tags in the user land.
But I don't think we would want to include that logic in an out-of-box support at the moment.

including the name of the pipeline in the email subject

yup. will make subject a func subject_fn then so the default includes pipeline name and users can overwrite it if that gets too sparse.

update subject -> subject_fn

subject updated so it defaults to group by pipeline name:

image.png (200×1 px, 48 KB)

This revision is now accepted and ready to land.Jun 25 2021, 11:23 PM
This revision was landed with ongoing or failed builds.Jun 26 2021, 12:40 AM
This revision was automatically updated to reflect the committed changes.