HomePhabricator

input manager

Description

input manager

Summary:
Additions:

  • A "manager_key" arg to InputDefinition, which allows users to give a resource that determines how the input is loaded.
  • A @input_manager decorator, which defines a resource that loads inputs using the decorated function.

Semantics:

  • If an input def has a manager key, it overrides other ways of loading the input. I.e. if input corresponds to an upstream output, the manager will be used instead of the asset store on that upstream output. If the input is unsatisfied, the manager will be used instead of the dagster_type_loader.
  • Input managers are configurable per-input. If an InputDefinition has a manager key, the manager is used instead of the dagster_type_loader to define the config schema for the input.
  • When loading an input that corresponds to an upstream output, the context passed to the loader function includes the step output handle and asset_metadata for the upstream.

Test Plan: bk

Reviewers: alangenfeld, schrockn, yuhan

Reviewed By: schrockn

Differential Revision: https://dagster.phacility.com/D5391

Details

Provenance
sandyryzaAuthored on Dec 2 2020, 4:19 PM
Reviewer
schrockn
Differential Revision
D5391: input manager
Parents
R1:47ee18cb31a9: Excise six.string_types
Branches
Unknown
Tags
Unknown