Page MenuHomePhabricator

Consolidate Airflow and Dask GraphQL client logic
ClosedPublic

Authored by natekupp on Jul 23 2019, 5:06 PM.

Details

Reviewers
max
Group Reviewers
Restricted Project
Commits
R1:66e054d96c91: Consolidate Airflow and Dask GraphQL client logic
Summary

This diff cleans up the divergent code paths that dagster-airflow and dagster-dask used to execute pipelines through dagster-graphql.

Now, a single function is provided in dagster-graphql, execute_start_pipeline_execution_query() to execute the pipelines and return a list of DagsterEvents.

This is in preparation for a following diff which will consume the dagster events to inform Airflow that a particular task was skipped using the Airflow SkipMixin.

Test Plan

unit

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

natekupp created this revision.Jul 23 2019, 5:06 PM
natekupp updated this revision to Diff 3096.Jul 23 2019, 5:34 PM

fix tests

natekupp updated this revision to Diff 3110.Jul 23 2019, 10:55 PM

remove unused files

natekupp edited the summary of this revision. (Show Details)Jul 23 2019, 10:58 PM
natekupp added a reviewer: Restricted Project.
max accepted this revision.Jul 24 2019, 6:01 PM
max added a subscriber: max.

oh, this is so great

python_modules/dagster-airflow/dagster_airflow_tests/test_dagster_docker_operator.py
62

hooray

This revision is now accepted and ready to land.Jul 24 2019, 6:01 PM