Page MenuHomePhabricator

Fix for Dagster errors not failing tasks in Airflow
ClosedPublic

Authored by nate on Dec 5 2019, 6:27 PM.

Details

Summary

This addresses the issue where Dagster step failures were not being surfaced as task failures in Airflow

Test Plan

unit, and manually tested a failing pipeline in Airflow UI with:

dagster-airflow scaffold -m dagster_examples.toys.airflow_examples --pipeline-name demo_error_pipeline
docker run -d -p 8080:8080 \
 -v $AIRFLOW_HOME/dags:/usr/local/airflow/dags dagster-airflow-webserver webserver

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

nate created this revision.Dec 5 2019, 6:27 PM
nate edited the test plan for this revision. (Show Details)Dec 5 2019, 6:30 PM
nate added reviewers: alangenfeld, max.
nate retitled this revision from Fix for Dagster errors failing tasks in Airflow to Fix for Dagster errors not failing tasks in Airflow.
alangenfeld added inline comments.Dec 5 2019, 6:43 PM
python_modules/dagster-airflow/dagster_airflow/operators/util.py
14–16

is AirflowException the right thing to throw? Should we do something else?

alangenfeld accepted this revision.Dec 5 2019, 6:51 PM
This revision is now accepted and ready to land.Dec 5 2019, 6:51 PM
nate added inline comments.Dec 5 2019, 6:52 PM
python_modules/dagster-airflow/dagster_airflow/operators/util.py
14–16

I can't seem to find this documented anywhere and conflicting advice given here https://stackoverflow.com/questions/43111506/how-do-i-force-a-task-on-airflow-to-fail for ValueError vs AirflowException, so went spelunking in the codebase - yes, this seems like the right thing to do to mark the task as failed:

https://github.com/apache/airflow/blob/a36cfe049a2c5948b24fde7a878fe19cabede5f7/airflow/models/taskinstance.py#L947-L958