Page MenuHomeElementl

fix partition run matrix to handle composite solids / dynamic solids
ClosedPublic

Authored by prha on Apr 9 2021, 1:22 AM.

Details

Summary

We were using solid / solid names to select and label step keys, which is incompatible with
composite solids and dynamic solids.

This diff reuses the PipelineExplorerRoot utility function explodeCompositesInHandleGraph, which
resolves solid handles to its flattened constituent targets, while remapping the component solid
names in all the dependency references.

The main change to the explodeCompositesInHandleGraph was that inputs/outputs specified on the
composite solids were not being transferred to its mapped constituent solids in this function.
This resulted in some dependencies being missed (not sure if this was a bug in the explorer view,
but I think it might have been for some subset of composition shapes).

For dynamic solids, we just rolled up any mapped result to its container solid, by loosely ignoring
any [mapping_key] suffix using a regex match

Before:

After:

Test Plan

added partition sets to dynamic_pipeline and composition toy pipelines, made sure the
list of solids in the matrix was flattened

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

prha requested review of this revision.Apr 9 2021, 1:29 AM
This revision is now accepted and ready to land.Apr 9 2021, 4:07 PM
js_modules/dagit/packages/core/src/partitions/useMatrixData.tsx
56–58
js_modules/dagit/packages/core/src/pipelines/PipelineExplorerRoot.tsx
107–109
113

Is it safe to mutate this object?

125–132

Same comments as above.

js_modules/dagit/packages/core/src/pipelines/PipelineExplorerRoot.tsx
113

yes, all these operations perform mutations.