Differential D5937

rename s3_io_manager to s3_pickled_io_manager, adls2_io_manager to adls2_pickled_io_manager, and gcs_io_manager to gcs_pickled_io_manager.
ClosedPublic
Actions

Authored by cdecarolis on Jan 12 2021, 9:13 PM.

Tags

None

Subscribers

Details

Reviewers

sandyryza
sashank
max
alangenfeld
yuhan

Commits

R1:569c87866245: rename s3_io_manager to s3_pickled_io_manager, adls2_io_manager to…

Summary

This diff renames the default io managers to their pickled counterparts. This change is worth doing for two reasons: 1. expose to the user what's actually happening under the hood, 2. Leave open the possibility of including a more generic io_manager for each of these storage options that performs serialization based on the dagster type.

Test Plan

Unit tests

Diff Detail

Repository

Branch

io_manager_rename

Lint

Lint OK

Unit

No Unit Test Coverage

Event Timeline

cdecarolis created this revision.Jan 12 2021, 9:13 PM

Harbormaster completed remote builds in B24187: Diff 29423.Jan 12 2021, 9:33 PM

cdecarolis requested review of this revision.Jan 12 2021, 9:33 PM

lg2m. but i don't feel strongly either way so i defer to people who hold stronger opinions :)

lgtm!

CC @schrockn in case he wants to weigh in.

oops meant to hit accept

This revision is now accepted and ready to land.Jan 13 2021, 1:49 AM

Closed by commit R1:569c87866245: rename s3_io_manager to s3_pickled_io_manager, adls2_io_manager to… (authored by Chris DeCarolis <cdguitar817@gmail.com>). · Explain WhyJan 13 2021, 2:49 PM

This revision was automatically updated to reflect the committed changes.

Chris DeCarolis <cdguitar817@gmail.com> added a commit: R1:569c87866245: rename s3_io_manager to s3_pickled_io_manager, adls2_io_manager to….

Revision Contents
Changeset List

Path

Size

python_modules/

libraries/

dagster-aws/

dagster_aws/

s3/

2 lines

9 lines

dagster_aws_tests/

s3_tests/

test_io_manager.py

8 lines

dagster-azure/

dagster_azure/

adls2/

2 lines

10 lines

dagster_azure_tests/

adls2_tests/

test_io_manager.py

8 lines

dagster-gcp/

dagster_gcp/

gcs/

2 lines

9 lines

dagster_gcp_tests/

gcs_tests/

test_io_manager.py

8 lines

Diff 29423

python_modules/libraries/dagster-aws/dagster_aws/s3/init.py

Loading...

python_modules/libraries/dagster-aws/dagster_aws/s3/io_manager.py

Loading...

python_modules/libraries/dagster-aws/dagster_aws_tests/s3_tests/test_io_manager.py

Loading...

python_modules/libraries/dagster-azure/dagster_azure/adls2/init.py

Loading...

python_modules/libraries/dagster-azure/dagster_azure/adls2/io_manager.py

Loading...

python_modules/libraries/dagster-azure/dagster_azure_tests/adls2_tests/test_io_manager.py

Loading...

python_modules/libraries/dagster-gcp/dagster_gcp/gcs/init.py

Loading...

python_modules/libraries/dagster-gcp/dagster_gcp/gcs/io_manager.py

Loading...

python_modules/libraries/dagster-gcp/dagster_gcp_tests/gcs_tests/test_io_manager.py

Loading...