HomePhabricator

DynamicOutput execution support

Description

DynamicOutput execution support

Summary:
The big one, this introduces all the Unresolved pieces to support dynamic unfolding and makes execution work.

Interesting things to note:

  • This takes the short cut of not giving the snapshot full information, it just gives it a representation of the deps with the unresolved steps, so using the [?] step key form
  • ExecutionPlan updates its structures as it resolves, couldn't really think of a better way to do this.
  • The execution result objects are solid handle oriented, and so this just takes the approach of collecting the mapped results in a simliar way to how we do on the step that does the original mapping. Not the most elegant but works.

Test Plan: added tests, will likely add more

Reviewers: catherinewu, schrockn

Reviewed By: schrockn

Differential Revision: https://dagster.phacility.com/D5542

Details

Provenance
alangenfeldAuthored on Dec 10 2020, 7:30 PM
Reviewer
schrockn
Differential Revision
D5542: DynamicOutput execution support
Parents
R1:5aaebe6e226d: DagsterEvent step_handle
Branches
Unknown
Tags
Unknown