Page MenuHomeElementl

dagstermill - fix: yield_result temp path includes solid name
ClosedPublic

Authored by yuhan on Jul 26 2021, 6:54 AM.

Details

Summary

fix a bug where yield_result wrote output value to the same file path if output names are the same for different solids.
it caused random serdes error in multiproc bc write and read could happen the same time for different objects to the same file

Test Plan

unit - fan_in_notebook_pipeline is the repro

before the fix, both solids yield "world" because one overwrites the other:

Screen Shot 2021-07-25 at 11.37.55 PM.png (776×2 px, 351 KB)

Diff Detail

Repository
R1 dagster
Branch
yuhan/notebook-output-path
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

yuhan added reviewers: alangenfeld, max.
yuhan edited the test plan for this revision. (Show Details)
Harbormaster returned this revision to the author for changes because remote builds failed.Jul 26 2021, 7:37 AM
Harbormaster failed remote builds in B34181: Diff 42250!
Harbormaster failed remote builds in B34180: Diff 42249!
yuhan requested review of this revision.Jul 26 2021, 6:16 PM

looking forward to a more thorough revamp of this stuff

This revision is now accepted and ready to land.Jul 27 2021, 4:13 PM