Page MenuHomePhabricator
Differential D5963

1/ Modernize airline_demo setup
ClosedPublic
Actions

Authored by max on Jan 13 2021, 7:10 PM.

Details

Reviewers
sandyryza
yuhan
Commits
R1:1ffcc61509d8: 1/ Modernize airline_demo setup
Summary

This moves the airline demo to use the new io_manager abstractions.

Test Plan

Unit

Diff Detail

Repository
R1 dagster
Branch
dagstermill-file-handling
Lint
Lint Errors
SeverityLocationCodeMessage
Errorexamples/airline_demo/airline_demo/solids.py:165E1123Unexpected Keyword Arg
Errorexamples/airline_demo/airline_demo/solids.py:206E1123Unexpected Keyword Arg
Errorexamples/airline_demo/airline_demo/solids.py:219E1123Unexpected Keyword Arg
Errorexamples/airline_demo/airline_demo/solids.py:262E1123Unexpected Keyword Arg
Errorexamples/airline_demo/airline_demo/solids.py:548E1123Unexpected Keyword Arg
Errorexamples/airline_demo/airline_demo_tests/unit_tests/test_load_data_from_spark.py:34E1123Unexpected Keyword Arg
Unit
No Unit Test Coverage

Event Timeline

max created this revision.Jan 13 2021, 7:10 PM
max updated this revision to Diff 29544.Jan 13 2021, 7:13 PM

rename

max updated this revision to Diff 29546.Jan 13 2021, 7:14 PM

rename

Harbormaster failed remote builds in B24282: Diff 29542!Jan 13 2021, 7:27 PM
Harbormaster failed remote builds in B24284: Diff 29544!Jan 13 2021, 7:31 PM
Harbormaster completed remote builds in B24286: Diff 29546.
max requested review of this revision.Jan 13 2021, 7:33 PM
max added a child revision: D5968: 2/ Modernize examples.Jan 13 2021, 7:40 PM
sandyryza accepted this revision.EditedJan 13 2021, 7:41 PM

Looks good! Left a comment that I think is worth addressing before merging.

examples/airline_demo/airline_demo/pipelines.py
88

You should be able to take out these intermediate_storage_def arguments

This revision is now accepted and ready to land.Jan 13 2021, 7:41 PM
max marked an inline comment as done.Jan 13 2021, 7:45 PM
max updated this revision to Diff 29556.Jan 13 2021, 7:45 PM

comments

max updated this revision to Diff 29559.Jan 13 2021, 7:56 PM

storage defs

Harbormaster failed remote builds in B24295: Diff 29556!Jan 13 2021, 8:04 PM
Harbormaster completed remote builds in B24298: Diff 29559.Jan 13 2021, 8:14 PM
Closed by commit R1:1ffcc61509d8: 1/ Modernize airline_demo setup (authored by max). · Explain WhyJan 13 2021, 8:23 PM
This revision was automatically updated to reflect the committed changes.
max added a commit: R1:1ffcc61509d8: 1/ Modernize airline_demo setup.

Revision Contents
Changeset List

Diff 29542

View Options

examples/airline_demo/airline_demo/pipelines.py

Loading...
View Options

examples/airline_demo/airline_demo/solids.py

Loading...
View Options

examples/airline_demo/airline_demo_tests/test_pipelines.py

Loading...
View Options

examples/airline_demo/airline_demo_tests/test_solids.py

Loading...
View Options

examples/airline_demo/airline_demo_tests/test_types.py

Loading...
View Options

examples/airline_demo/airline_demo_tests/unit_tests/test_cache_file_from_s3.py

Loading...
View Options

examples/airline_demo/airline_demo_tests/unit_tests/test_ingest_csv_file_handle_to_spark.py

Loading...
View Options

examples/airline_demo/airline_demo_tests/unit_tests/test_load_data_from_spark.py

Loading...
View Options

examples/airline_demo/airline_demo_tests/unit_tests/test_unzip_file_handle.py

Loading...
View Options

examples/airline_demo/airline_demo_tests/unit_tests/testing_guide_tests/test_cache_file_from_s3_in_guide_step_four.py

Loading...
View Options

examples/airline_demo/airline_demo_tests/unit_tests/testing_guide_tests/test_cache_file_from_s3_in_guide_step_three.py

Loading...
View Options

examples/airline_demo/an_archive_member

Loading...
View Options

examples/airline_demo/setup.py

Loading...