Page MenuHomePhabricator
Differential D4745

Don't load pyspark when importing dagster_aws.emr
AbandonedPublic
Actions

Authored by sandyryza on Oct 9 2020, 5:52 PM.

Details

Reviewers
nate
cdecarolis
yuhan
schrockn
Summary

This is a breaking change for anyone relying on the emr_pyspark_step_launcher. Now they need to do
from dagster_aws.emr.pyspark import emr_pyspark_step_launcher instead of
from dagster_aws.emr import emr_pyspark_step_launcher.

Test Plan

bk

Diff Detail

Repository
R1 dagster
Branch
pyspark_step_launcher (branched from master)
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

sandyryza created this revision.Oct 9 2020, 5:52 PM
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 9 2020, 6:10 PM
Harbormaster failed remote builds in B19373: Diff 23543!
sandyryza updated this revision to Diff 23544.Oct 9 2020, 6:15 PM

up

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 9 2020, 6:33 PM
Harbormaster failed remote builds in B19374: Diff 23544!
sandyryza updated this revision to Diff 23547.Oct 9 2020, 6:44 PM

up

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 9 2020, 7:12 PM
Harbormaster failed remote builds in B19376: Diff 23547!
sandyryza updated this revision to Diff 23556.Oct 9 2020, 8:51 PM

up

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 9 2020, 9:07 PM
Harbormaster failed remote builds in B19384: Diff 23556!
sandyryza updated this revision to Diff 23559.Oct 9 2020, 9:34 PM

up

Harbormaster completed remote builds in B19386: Diff 23559.Oct 9 2020, 9:51 PM
sandyryza requested review of this revision.Oct 9 2020, 9:51 PM
schrockn requested changes to this revision.Oct 13 2020, 3:12 PM
schrockn added a subscriber: schrockn.

As long we are doing a proper breaking change, why not move into its own module and actually model the dependencies properly?

This revision now requires changes to proceed.Oct 13 2020, 3:12 PM
sandyryza abandoned this revision.Oct 20 2020, 1:14 AM

Superseded by https://dagster.phacility.com/D4806

Revision Contents
Changeset List

PathSize
docs/
sections/
api/
apidocs/
libraries/
2 lines
examples/
airline_demo/
airline_demo/
2 lines
emr_pyspark/
2 lines
legacy_examples/
dagster_examples/
simple_pyspark/
2 lines
python_modules/
libraries/
dagster-aws/
dagster_aws/
emr/
1 line
pyspark.py
pyspark_step_launcher.py
341 lines
dagster_aws_tests/
emr_tests/
8 lines
4 lines

Diff 23559

View Options

docs/sections/api/apidocs/libraries/dagster_aws.rst

Loading...
View Options

examples/airline_demo/airline_demo/pipelines.py

Loading...
View Options

examples/emr_pyspark/repo.py

Loading...
View Options

examples/legacy_examples/dagster_examples/simple_pyspark/pipelines.py

Loading...
View Options

python_modules/libraries/dagster-aws/dagster_aws/emr/__init__.py

Loading...
View Options

python_modules/libraries/dagster-aws/dagster_aws/emr/pyspark.py

Loading...
View Options

python_modules/libraries/dagster-aws/dagster_aws/emr/pyspark_step_launcher.py

Loading...
View Options

python_modules/libraries/dagster-aws/dagster_aws_tests/emr_tests/test_pyspark.py

Loading...
View Options

python_modules/libraries/dagster-aws/dagster_aws_tests/emr_tests/test_pyspark_step_launcher.py

Loading...