UI support for dynamic outputs, Run view state cleanup


UI support for dynamic outputs, Run view state cleanup

This diff exposes the "isDynamic" property on the OutputDefinition GraphQL interface, and uses it in the Pipeline > Overview tab to show a "1-to-many" (*) on outputs that are dynamic on the DAG view and the sidebar.

The Run details page has been refactored quite a bit - previously, this view pulled in the execution plan for the run and the run logs, but they were used side by side. In order to support dynamic outputs, we need to assemble the metadata from the logs (eg: step keys), and transform the execution plan to reflect the actual invocations of the dynamic steps (by duplicating unresolved sub-graphs).

In addition to shifting the data around a bit, I also had to change the way StepSelection is persisted on the Run page. Previously we kept both the user input "query" and the resolved "step keys" in state. Because the query => keys transform is dependent on the keys discovered in the logs at runtime it's now computed rather than stored. I think this actually made a bunch of stuff cleaner!

I also updated the Run tab to useQueryPersistedState for the page state instead of the bespoke url->state and copy-to-clipboard functionality. I think the old implementation used URLSearchParams nicely though and in a separate diff we may want to explore using that class within the standard hook to serialize array values into the query string more cleanly.

There are a few places Dagit is still pulling apart step keys and relying on the [?] style syntax. I think this is actually unavoidable because it needs to transform the execution plan based on the step keys it discovers at runtime and needs to understand how to do the replacement. To keep things as clean as possible though, this all runs through helper functions in DynamicStepSupport.tsx.

image.png (324×534 px, 30 KB)

image.png (208×296 px, 13 KB)

image.png (1×1 px, 240 KB)

Test Plan: Run new tests!

Reviewers: schrockn, dish, alangenfeld

Reviewed By: dish

Differential Revision: https://dagster.phacility.com/D6132


bengotowAuthored on Jan 23 2021, 8:05 PM
Differential Revision
D6132: UI support for dynamic outputs, Run view state cleanup
R1:df93cfde6c4e: Flip asset materialization UI so newest time/partition is on the left