Page MenuHomePhabricator

[prototype] map / fan-out
Changes PlannedPublicDraft

Authored by catherinewu on Oct 20 2020, 4:39 PM.

Details

Reviewers
None
Summary

Forked Alex's D4259 w/ https://excalidraw.com/#json=4672312762695680,1d7FSwy6YBAKBp-wDhmjGA

known issues:

  • re-visit usage "special" -- perhaps "mappable"?
  • re-execute from failure doesn't work yet{F359104}
    • update: re-execute from failure hackily works. still need to persist parent's "unresolved" execution plan, load the "unresolved" execution plan, and walk through events while updating the execution plan
  • solids receiving a mapped input cannot receive any other input
  • gaant chart doesnt display anything
Test Plan

elmofire

Diff Detail

Repository
R1 dagster
Branch
arcpatch-D4831
Lint
Lint WarningsExcuse: .
SeverityLocationCodeMessage
Warningpython_modules/dagster-test/dagster_test/toys/map_pipe.py:7W0613Unused Argument
Warningpython_modules/dagster/dagster/core/execution/memoization.py:62W0002Print Call
Warningpython_modules/dagster/dagster/core/execution/memoization.py:73W0002Print Call
Warningpython_modules/dagster/dagster/core/execution/memoization.py:75W0002Print Call
Warningpython_modules/dagster/dagster/core/execution/memoization.py:79W0002Print Call
Warningpython_modules/dagster/dagster/core/execution/memoization.py:82W0002Print Call
Warningpython_modules/dagster/dagster/core/execution/memoization.py:88W0002Print Call
Warningpython_modules/dagster/dagster/core/execution/plan/objects.py:395W0406Import Self
Warningpython_modules/dagster/dagster/core/execution/plan/objects.py:395W0621Redefined Outer Name
Warningpython_modules/dagster/dagster/core/storage/intermediate_storage.py:199W0002Print Call
Warningpython_modules/dagster/dagster_tests/core_tests/execution_tests/test_map.py:1W0611Unused Import
Warningpython_modules/dagster/dagster_tests/core_tests/execution_tests/test_map.py:2W0611Unused Import
Warningpython_modules/dagster/dagster_tests/core_tests/execution_tests/test_map.py:4W0611Unused Import
Warningpython_modules/dagster/dagster_tests/core_tests/execution_tests/test_map.py:4W0611Unused Import
Warningpython_modules/dagster/dagster_tests/core_tests/execution_tests/test_map.py:4W0611Unused Import
Warningpython_modules/dagster/dagster_tests/core_tests/execution_tests/test_map.py:19W0611Unused Import
Warningpython_modules/dagster/dagster_tests/core_tests/execution_tests/test_map.py:67W0621Redefined Outer Name
Warningpython_modules/dagster/dagster_tests/core_tests/execution_tests/test_map.py:90W0621Redefined Outer Name
Warningpython_modules/dagster/dagster_tests/core_tests/execution_tests/test_map.py:98W0613Unused Argument
Warningpython_modules/dagster/dagster_tests/core_tests/execution_tests/test_map.py:132W0613Unused Argument
Warningpython_modules/dagster/dagster_tests/core_tests/execution_tests/test_map.py:132W0621Redefined Outer Name
Unit
No Unit Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 20 2020, 4:59 PM
Harbormaster failed remote builds in B19843: Diff 24073!
catherinewu edited the summary of this revision. (Show Details)

super hacky but working re-execute from failure

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 22 2020, 1:48 AM
Harbormaster failed remote builds in B19981: Diff 24247!

make re-execution work for solids that are downstream of the mapped solid

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 22 2020, 8:06 PM
Harbormaster failed remote builds in B20025: Diff 24294!
catherinewu edited the summary of this revision. (Show Details)

add some tests. can't find the test utility to reexecute pipeline from failure

this diff will hash out parent run execution plan serdes + subsequent traversal

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 22 2020, 10:50 PM
Harbormaster failed remote builds in B20049: Diff 24327!