Page MenuHomePhabricator

[dagster] error on unresolved inputs that cant be hydrated
ClosedPublic

Authored by alangenfeld on Jul 31 2019, 10:53 PM.

Details

Reviewers
max
Group Reviewers
Restricted Project
Commits
R1:10175b807be7: [dagster] error on unresolved inputs that cant be hydrated
Summary

This diff resolves an issue where you could create unexecutable pipelines by selecting subsets where the dangling inputs can not be materialized from config. To solve this we:

  • introduce validation at pipeline instantiation time that all unresolved inputs must have a hydration config
  • create a single InvalidSubsetError type that incapsulates both bad solid names as well as the issue described above
  • ensure that the UI communicates these problems effectively when selecting invalid subsets
Test Plan

new test case

load up the airline demo ingest pipeline and select subsets that are invalid

Diff Detail

Repository
R1 dagster
Branch
invalid-subset (branched from master)
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

alangenfeld created this revision.Jul 31 2019, 10:53 PM

skip over Nothing

alangenfeld updated this revision to Diff 3358.Aug 1 2019, 3:17 PM

make Stringish use string hydration/materialization

Harbormaster failed remote builds in B2676: Diff 3359!
alangenfeld updated this revision to Diff 3360.Aug 1 2019, 6:52 PM

attempt to fix issue

alangenfeld updated this revision to Diff 3362.Aug 1 2019, 8:18 PM

fix types test

alangenfeld updated this revision to Diff 3374.Aug 2 2019, 5:33 PM

handle UI

alangenfeld planned changes to this revision.Aug 2 2019, 6:45 PM
alangenfeld edited the summary of this revision. (Show Details)
alangenfeld edited the test plan for this revision. (Show Details)
alangenfeld added a reviewer: Restricted Project.
alangenfeld updated this revision to Diff 3381.Aug 2 2019, 11:35 PM

clean up pass

max added a subscriber: max.Aug 6 2019, 7:05 PM

Maybe add a note to mark this error message -- eventually we will want to link to the docs, e.g. https://dagster.readthedocs.io/en/0.5.5/sections/learn/tutorial/types.html#input-hydration-config

max accepted this revision.Aug 6 2019, 7:07 PM
This revision is now accepted and ready to land.Aug 6 2019, 7:07 PM