HomePhabricator

add hooks to route standard out/err from a compute step to a file

Authored by prha on Tue, Aug 27, 11:07 PM.

Description

add hooks to route standard out/err from a compute step to a file

Summary:
Creates a contextmanager to wrap any compute function execution and
routes all streams to the filesystem if configured. By default, we store all
stdout/stderr logs to a subdirectory within the $DAGSTER_HOME directory. All of
the existing behavior is maintained if $DAGSTER_HOME is not set.

Test Plan:
Adds a toy example that allows both the multiprocess and inprocess
execution engines to pipe stdout/stderr to disk.

Reviewers: #ft, natekupp

Reviewed By: #ft, natekupp

Subscribers: alangenfeld, natekupp

Differential Revision: https://dagster.phacility.com/D893

Details

Committed
prhaThu, Aug 29, 12:04 AM
Reviewer
Restricted Project
Differential Revision
D893: add hooks to route standard out/err from a compute step to a file
Parents
R1:bb85c68508b4: Remove lambda_solid from tutorial and examples
Branches
Unknown
Tags
Unknown