HomeElementl

fix partition run matrix to handle composite solids / dynamic solids

Description

fix partition run matrix to handle composite solids / dynamic solids

Summary:
We were using solid / solid names to select and label step keys, which is incompatible with
composite solids and dynamic solids.

This diff reuses the PipelineExplorerRoot utility function explodeCompositesInHandleGraph, which
resolves solid handles to its flattened constituent targets, while remapping the component solid
names in all the dependency references.

The main change to the explodeCompositesInHandleGraph was that inputs/outputs specified on the
composite solids were not being transferred to its mapped constituent solids in this function.
This resulted in some dependencies being missed (not sure if this was a bug in the explorer view,
but I think it might have been for some subset of composition shapes).

For dynamic solids, we just rolled up any mapped result to its container solid, by loosely ignoring
any [mapping_key] suffix using a regex match

Before:

After:

Test Plan:
added partition sets to dynamic_pipeline and composition toy pipelines, made sure the
list of solids in the matrix was flattened

Reviewers: alangenfeld, dish, sandyryza

Reviewed By: alangenfeld

Differential Revision: https://dagster.phacility.com/D7365

Details

Provenance
prhaAuthored on Apr 9 2021, 12:08 AM
Reviewer
alangenfeld
Differential Revision
D7365: fix partition run matrix to handle composite solids / dynamic solids
Parents
R1:e2617ae70897: K8sRunLauncher make postgres optional
Branches
Unknown
Tags
Unknown