Page MenuHomeElementl
Differential D7160

add compute log manager api to more directly expose file-like object
AbandonedPublic
Actions

Authored by • prha on Mar 26 2021, 4:57 PM.
Tags
None
Referenced Files
Unknown Object (File)
Tue, Aug 9, 11:05 AM
Unknown Object (File)
Sun, Aug 7, 8:48 AM
Unknown Object (File)
Sun, Aug 7, 7:13 AM
Unknown Object (File)
Mon, Aug 1, 8:20 AM
Unknown Object (File)
Sat, Jul 30, 8:18 AM
Unknown Object (File)
Thu, Jul 28, 6:22 PM
Unknown Object (File)
Fri, Jul 22, 11:18 AM
Unknown Object (File)
Fri, Jul 22, 8:09 AM
View All 38 Files
Subscribers
None

Details

Reviewers
• max
• johann
alangenfeld
Summary

In follow-up diff, might want to remove the read_logs_file method on the compute log
manager in favor of these simpler file-like APIs (may require separate API for binary vs text).

Test Plan

bk

Diff Detail

Repository
R1 dagster
Branch
prha/compute_log
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

• prha created this revision.Mar 26 2021, 4:57 PM
Harbormaster completed remote builds in B28048: Diff 34380.Mar 26 2021, 5:22 PM
• prha requested review of this revision.Mar 26 2021, 5:22 PM
alangenfeld added a comment.Mar 26 2021, 6:10 PM

what was the motivation for this?

alangenfeld requested changes to this revision.Mar 26 2021, 6:11 PM

probably worth having tests for the 3 integrations

This revision now requires changes to proceed.Mar 26 2021, 6:11 PM
• prha added a comment.Mar 26 2021, 7:54 PM

Motivation I think might be related to commercialization? https://github.com/dagster-io/dagster/issues/3935

• prha updated this revision to Diff 34494.Mar 29 2021, 5:55 PM
  • add library tests
alangenfeld resigned from this revision.Mar 29 2021, 5:57 PM

k ill let @max sign off since he wrote that issue

python_modules/dagster/dagster/core/instance/__init__.py
522

hi

1559

^

Harbormaster failed remote builds in B28145: Diff 34494!Mar 29 2021, 7:00 PM
• prha updated this revision to Diff 34559.Mar 29 2021, 10:13 PM

rebase, kick off new bk

Harbormaster completed remote builds in B28191: Diff 34559.Mar 29 2021, 10:34 PM
• max accepted this revision.May 26 2021, 11:06 PM

thank you, this is great

This revision is now accepted and ready to land.May 26 2021, 11:06 PM
• prha added a reviewer: • johann.Jun 8 2021, 6:10 PM
• prha abandoned this revision.Jun 8 2021, 8:25 PM

going to abandon in favor of a new ProcessLogManager API or something similar.

Revision Contents
Changeset List

PathSize
python_modules/
dagster/
dagster/
core/
instance/
9 lines
storage/
20 lines
20 lines
6 lines
dagster_tests/
core_tests/
test_compute_logs.py
test_stdout.py
22 lines
libraries/
dagster-aws/
dagster_aws/
s3/
22 lines
dagster_aws_tests/
s3_tests/
46 lines
dagster-azure/
dagster_azure/
blob/
22 lines
dagster_azure_tests/
blob_tests/
59 lines
dagster-gcp/
dagster_gcp/
gcs/
22 lines
dagster_gcp_tests/
gcs_tests/
44 lines

Diff 34559

View Options

python_modules/dagster/dagster/core/instance/__init__.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/compute_log_manager.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/local_compute_log_manager.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/noop_compute_log_manager.py

Loading...
View Options

python_modules/dagster/dagster_tests/core_tests/test_compute_logs.py

Loading...
View Options

python_modules/dagster/dagster_tests/core_tests/test_stdout.py

Loading...
View Options

python_modules/libraries/dagster-aws/dagster_aws/s3/compute_log_manager.py

Loading...
View Options

python_modules/libraries/dagster-aws/dagster_aws_tests/s3_tests/test_compute_log_manager.py

Loading...
View Options

python_modules/libraries/dagster-azure/dagster_azure/blob/compute_log_manager.py

Loading...
View Options

python_modules/libraries/dagster-azure/dagster_azure_tests/blob_tests/test_compute_log_manager.py

Loading...
View Options

python_modules/libraries/dagster-gcp/dagster_gcp/gcs/compute_log_manager.py

Loading...
View Options

python_modules/libraries/dagster-gcp/dagster_gcp_tests/gcs_tests/test_compute_log_manager.py

Loading...