HomePhabricator

Rewrite of the Composite Config Mapping and Parsing Stack

Authored by schrockn on Dec 12 2019, 2:35 PM.

Description

Rewrite of the Composite Config Mapping and Parsing Stack

Summary:
This is a substantial rewrite of the config mapping and parsing
stack.

The big conceptual change is that we have separated our config
processing into to two separate steps that can be cleanly composed,
and, most importantly, invoked recursively in a way with complexity
more managed.

We now have a core config type system that does not know anything
about the config mappings and the notion of solids at all. It is
just pure schema.

The manipulations around config mapping the descent into composite
solid definitions is now managed in a totally different pass,
currently called "composite_descent". It's only job in life is
to recurse down the config tree and construct a dictionary from
solid handles to SolidConfigs.

When it encouters a mapping function it dynamically constructs (this
should be cached) a config type that we expect to be returned from
the config mapping function, and this is validated using the
validate_config machinery that knows *nothing* about config mapping
construct at all. From the standpoint of that machinery, it is only
the *interface* (that is, the declared config field) into the config
mapping that is relevant.

The way to manage this in your is to note that there will be at minimum
1 validate_config call in a complete pass. And then there will be N
additional calls for the N instances of solids that are from
composite definitions with config mapping functions.

Test Plan: BK

Reviewers: alangenfeld, nate

Reviewed By: alangenfeld

Differential Revision: https://dagster.phacility.com/D1649