Page MenuHomePhabricator

Handle AirflowSkipException without crashing
Needs RevisionPublic

Authored by catherinewu on May 7 2020, 1:23 AM.

Details

Reviewers
schrockn
nate
Summary

Catch AirflowSkipException and allow the solid to return successfully

Test Plan

Added

Diff Detail

Repository
R1 dagster
Branch
skip-operator
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

python_modules/libraries/dagster-airflow/dagster_airflow/dagster_pipeline_factory.py
438

hmm can we warn here or log an event or something, just so there's some record that we handled this?

python_modules/libraries/dagster-airflow/dagster_airflow_tests/test_dagster_pipeline_factory/test_solid_execution.py
228

prob don't need ui_color for the test :)

nate requested changes to this revision.May 7 2020, 3:44 AM

pushing back to your queue for above

This revision now requires changes to proceed.May 7 2020, 3:44 AM

rough draft, also needs tests

Process-wise I'd like for the SkipEvent to be added to the core on its own (without touching airflow) and with use cases and rationale clearly laid out. It should make sense in the absence of Airflow. Adding a top level event type to the system is a whole affair which needs to properly tested, thought through, and flowed down to the graphql layer.

This revision now requires changes to proceed.May 20 2020, 1:10 PM
python_modules/dagster/dagster/cli/pipeline.py
149–166

would just submit this change in a separate diff