In and Out

This aims to make input and output-specifications for ops briefer.

  • Adds an "ins" argument that accepts "In"s, instead of input_defs/InputDefinition
  • Adds an "out" argument that accepts either an "Out" or a "MultiOut", instead of output_defs/OutputDefinition.
  • For solids/ops with multiple output definitions or MultiOuts, allows the decorated function to provide a tuple that will be unpacked into multiple outputs. This reduces the set of cases where someone needs to yield an Output and makes it possible to annotate distinct Python types for each output.

Some ideas I considered that I think would be backwards-compatible to implement later if we wanted to:

  • Accept a dict of Ins instead of a List. I went in thinking I would do this, but came out not thinking it was particularly better. The number of characters a user needs to type is identical. It doesn't work for Outs, because, if we want the tuple-stuff mentioned above to work, the Outs need to be ordered, and supplying an OrderedDict is a pain.
  • Accept a list of Ins without names and do magical auto-matching with the positional inputs.
  • Accept a list of Outs without names and generate names like "result_1", "result_2".

Test Plan: bk

Reviewers: alangenfeld, schrockn, sandyryza

Reviewed By: alangenfeld

Differential Revision: https://dagster.phacility.com/D8435


Chris DeCarolis <cdguitar817@gmail.com>Authored on Jun 18 2021, 4:59 PM
Differential Revision
D8435: In and Out
R1:fa16b017596c: rename StoredEventRecord -> EventLogRecord

Event Timeline

Chris DeCarolis <cdguitar817@gmail.com> committed R1:344604bf7a1b: In and Out (authored by Chris DeCarolis <cdguitar817@gmail.com>).Jun 18 2021, 6:58 PM