Page MenuHomePhabricator

Fix for Dagster errors not failing tasks in Airflow

Authored by nate on Dec 5 2019, 6:27 PM.



This addresses the issue where Dagster step failures were not being surfaced as task failures in Airflow

Test Plan

unit, and manually tested a failing pipeline in Airflow UI with:

dagster-airflow scaffold -m --pipeline-name demo_error_pipeline
docker run -d -p 8080:8080 \
 -v $AIRFLOW_HOME/dags:/usr/local/airflow/dags dagster-airflow-webserver webserver

Diff Detail

R1 dagster
Automatic diff as part of commit; lint not applicable.
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

nate created this revision.Dec 5 2019, 6:27 PM
nate edited the test plan for this revision. (Show Details)Dec 5 2019, 6:30 PM
nate added reviewers: alangenfeld, max.
nate retitled this revision from Fix for Dagster errors failing tasks in Airflow to Fix for Dagster errors not failing tasks in Airflow.
alangenfeld added inline comments.Dec 5 2019, 6:43 PM

is AirflowException the right thing to throw? Should we do something else?

alangenfeld accepted this revision.Dec 5 2019, 6:51 PM
This revision is now accepted and ready to land.Dec 5 2019, 6:51 PM
nate added inline comments.Dec 5 2019, 6:52 PM

I can't seem to find this documented anywhere and conflicting advice given here for ValueError vs AirflowException, so went spelunking in the codebase - yes, this seems like the right thing to do to mark the task as failed: