Page MenuHomePhabricator

[log sink] multiprocess executor
ClosedPublic

Authored by alangenfeld on Aug 13 2019, 10:53 PM.

Details

Summary

use newly setup log sink to communicate logs from child processes to parent.

Test Plan

execute sleepy pipeline with multi proc can see info logs in dagit.

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

alangenfeld created this revision.Aug 13 2019, 10:53 PM
max accepted this revision.Aug 13 2019, 11:07 PM
max added inline comments.
python_modules/dagster/dagster/core/engine/engine_multiprocess.py
18

i'm not thrilled by adding more parameters to this

This revision is now accepted and ready to land.Aug 13 2019, 11:07 PM
alangenfeld updated this revision to Diff 3690.Aug 14 2019, 4:12 PM

try/finally & faster poll rate

alangenfeld updated this revision to Diff 3703.Aug 14 2019, 6:41 PM

open connection every write

alangenfeld updated this revision to Diff 3704.Aug 14 2019, 6:42 PM

rm spurious changes

alangenfeld updated this revision to Diff 3708.Aug 14 2019, 8:12 PM

cant remove the log_step_event call

This revision was automatically updated to reflect the committed changes.