Differential D812

[log sink] multiprocess executor
ClosedPublic
Actions

Authored by alangenfeld on Tue, Aug 13, 10:53 PM.

Tags

None

Subscribers

None

Details

Reviewers

Commits

R1:fb384ad3b262: [log sink] multiprocess executor

Summary

use newly setup log sink to communicate logs from child processes to parent.

Test Plan

execute sleepy pipeline with multi proc can see info logs in dagit.

Diff Detail

Repository

Branch

mplog (branched from master)

Lint

Lint OK

Unit

No Unit Test Coverage

Event Timeline

alangenfeld created this revision.Tue, Aug 13, 10:53 PM

alangenfeld added a parent revision: D811: [log sink] RunConfig log_sink.Tue, Aug 13, 10:55 PM

max accepted this revision.Tue, Aug 13, 11:07 PM

max added inline comments.

python_modules/dagster/dagster/core/engine/engine_multiprocess.py
18	i'm not thrilled by adding more parameters to this

This revision is now accepted and ready to land.Tue, Aug 13, 11:07 PM

Harbormaster failed remote builds in B2939: Diff 3672!Tue, Aug 13, 11:59 PM

rebase

lint

Harbormaster failed remote builds in B2948: Diff 3683!Wed, Aug 14, 3:11 PM

Harbormaster failed remote builds in B2950: Diff 3685!Wed, Aug 14, 3:18 PM

try/finally & faster poll rate

Harbormaster failed remote builds in B2954: Diff 3690!Wed, Aug 14, 4:40 PM

open connection every write

rm spurious changes

Harbormaster failed remote builds in B2965: Diff 3703!Wed, Aug 14, 6:46 PM

Harbormaster failed remote builds in B2966: Diff 3704!Wed, Aug 14, 6:49 PM

fixes

Harbormaster completed remote builds in B2967: Diff 3707.Wed, Aug 14, 7:31 PM

cant remove the log_step_event call

Harbormaster completed remote builds in B2968: Diff 3708.Wed, Aug 14, 8:25 PM

Closed by commit R1:fb384ad3b262: [log sink] multiprocess executor (authored by alangenfeld). · Explain WhyWed, Aug 14, 8:41 PM

This revision was automatically updated to reflect the committed changes.

alangenfeld added a commit: R1:fb384ad3b262: [log sink] multiprocess executor.

Revision Contents
Changeset List

			Path	Packages
M			examples/dagster_examples/toys/sleepy.py (7 lines)
M			python_modules/dagster/dagster/core/engine/engine_multiprocess.py (43 lines)
M			python_modules/dagster/dagster/core/events/log.py (3 lines)
M			python_modules/dagster/dagster/core/execution/context_creation_pipeline.py (1 line)
M			python_modules/dagster/dagster/loggers/xproc_log_sink.py (11 lines)
M			python_modules/dagster/dagster/utils/log.py (6 lines)

Diff 3683

examples/dagster_examples/toys/sleepy.py

Loading...

python_modules/dagster/dagster/core/engine/engine_multiprocess.py

Loading...

python_modules/dagster/dagster/core/events/log.py

Loading...

python_modules/dagster/dagster/core/execution/context_creation_pipeline.py

Loading...

python_modules/dagster/dagster/loggers/xproc_log_sink.py

Loading...

python_modules/dagster/dagster/utils/log.py

Loading...