HomePhabricator

Eliminate selector argument from PipelineDefinition

Authored by schrockn on Nov 27 2019, 11:06 PM.

Description

Eliminate selector argument from PipelineDefinition

Summary:
This argument is actually unnecessary because the selector property
can be derived from the solid definitions and the dependency graph. The
only code path that populates this is through build_sub_pipeline and that
codepath subsets the dependencies and the solid definitions. From this
information you can construct the set of solid instances that will execute

The current PipelineDefinition allowed for the pipeline to be constructed
in an invalid state, where the ExecutionSelection can potentially "lie" about
what will be executed, or vice versa.

This state of the world might be a bit awkward. (The constructing
a new PipelineDefinition that is a subset has always been a bit strange)

However post this diff the world is more consistent and the API
less error-prone

Test Plan: BK

Reviewers: alangenfeld, prha

Reviewed By: prha

Differential Revision: https://dagster.phacility.com/D1501

Details

Committed
schrocknDec 2 2019, 5:28 PM
Reviewer
prha
Differential Revision
D1501: Eliminate selector argument from PipelineDefinition
Parents
R1:2d85b76fb77b: [airflow] fix error bug
Branches
Unknown
Tags
Unknown