HomePhabricator

(make-pipeline-a-solid-1) Allow pipeline to be used as composite_solid

Description

(make-pipeline-a-solid-1) Allow pipeline to be used as composite_solid

Summary:
This is committable, but it is also a larger diff than I
generally do for this type of thing, so I could easily be persuaded to
break it up.

In order to reconcile composite_solid and pipeline into graph, we
have to take a piecemeal approach.

What this diff does is introduce a new base class GraphDefinition
that both CompositeSolidDefinition and PipelineDefinition inherit
from. The eventual long-term goal would be a delete both CompositeSolidDefinition
and PipelineDefinition, with only GraphDefinition surviving. The
0.10.0 goal would be to have CompositeSolidDefinition and
PipelineDefiniton be trivial subclasses, with only backcompat (e.g.
argument ordering) issues involved.

The net result of this diff is that you can use @pipeline
anywhere you can use @composite_solid. E.g. you can embed
pipelines in pipelines, making it a unit of composition.

Next steps would be to make the config system work with top-level
config, inputs, and outputs keys. Right now if you tried to
use execute_pipeline on a pipeline with a config mapping or an
unfulfilled input it would break.

Test Plan: BK

Reviewers: alangenfeld, max, dgibson, sandyryza

Reviewed By: alangenfeld

Differential Revision: https://dagster.phacility.com/D4847