Details

Reviewers

schrockn
• nate

Commits

R1:0bfc6665cd76: create dagster-aws-pyspark library with emr_pyspark_step_launcher

Test Plan

bk, pip install

Diff Detail

Repository

R1 dagster

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

sandyryza created this revision.Oct 16 2020, 9:00 PM

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 16 2020, 9:18 PM

Harbormaster failed remote builds in B19737: Diff 23956!

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 16 2020, 10:06 PM

Harbormaster failed remote builds in B19742: Diff 23961!

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 16 2020, 11:18 PM

Harbormaster failed remote builds in B19744: Diff 23963!

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 4:15 PM

Harbormaster failed remote builds in B19756: Diff 23977!

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 6:11 PM

Harbormaster failed remote builds in B19771: Diff 23991!

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 7:04 PM

Harbormaster failed remote builds in B19778: Diff 23999!

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 8:48 PM

Harbormaster failed remote builds in B19797: Diff 24020!

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 10:15 PM

Harbormaster failed remote builds in B19810: Diff 24034!

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 11:05 PM

Harbormaster failed remote builds in B19815: Diff 24040!

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 20 2020, 12:25 AM

Harbormaster failed remote builds in B19819: Diff 24045!

Harbormaster completed remote builds in B19826: Diff 24053.Oct 20 2020, 1:08 AM

sandyryza requested review of this revision.Oct 20 2020, 1:08 AM

sandyryza mentioned this in D4745: Don't load pyspark when importing dagster_aws.emr.Oct 20 2020, 1:14 AM

should we call it dagster-emr?

This revision now requires changes to proceed.Oct 23 2020, 10:45 PM

should we call it dagster-emr?

dagster-aws has some non-PySpark EMR stuff in it (EmrJobRunner). The motivation for this change came from a user who wanted to be able to use that stuff without pulling in pyspark dependencies.

makes sense

This revision is now accepted and ready to land.Oct 24 2020, 12:46 AM

Closed by commit R1:0bfc6665cd76: create dagster-aws-pyspark library with emr_pyspark_step_launcher (authored by sandyryza). · Explain WhyOct 26 2020, 3:14 PM

This revision was automatically updated to reflect the committed changes.

sandyryza added a commit: R1:0bfc6665cd76: create dagster-aws-pyspark library with emr_pyspark_step_launcher.

sandyryza added a reverting change: D4940: Revert dagster-aws-pyspark.Oct 28 2020, 4:48 PM

sandyryza added a reverting change: R1:696fd43cb149: Revert dagster-aws-pyspark.Oct 29 2020, 6:32 PM

create dagster-aws-pyspark library with emr_pyspark_step_launcher
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 24432

.buildkite/pipeline.py

Makefile

docs-requirements.txt

docs/next/src/pages/overview/packages/libraries.mdx

docs/sections/api/apidocs/libraries/dagster_aws.rst

docs/sections/api/apidocs/libraries/dagster_aws_pyspark.rst

examples/airline_demo/airline_demo/pipelines.py

examples/airline_demo/setup.py

examples/airline_demo/tox.ini

examples/emr_pyspark/repo.py

examples/emr_pyspark/requirements.txt

examples/emr_pyspark/tox.ini

examples/legacy_examples/Dockerfile

examples/legacy_examples/dagster_examples/simple_pyspark/pipelines.py

examples/legacy_examples/setup.py

examples/legacy_examples/tox.ini

python_modules/libraries/dagster-aws-pyspark/.coveragerc

python_modules/libraries/dagster-aws-pyspark/README.md

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark/init.py

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark/pyspark_step_launcher.py

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark/version.py

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark_tests/init.py

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark_tests/test_pyspark_step_launcher.py

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark_tests/test_version.py

python_modules/libraries/dagster-aws-pyspark/dev-requirements.txt

python_modules/libraries/dagster-aws-pyspark/requirements.txt

python_modules/libraries/dagster-aws-pyspark/setup.py

python_modules/libraries/dagster-aws-pyspark/tox.ini

python_modules/libraries/dagster-aws/dagster_aws/emr/init.py

python_modules/libraries/dagster-aws/dagster_aws/emr/pyspark_step_launcher.py

python_modules/libraries/dagster-aws/dagster_aws_tests/emr_tests/test_pyspark.py

python_modules/libraries/dagster-aws/dagster_aws_tests/emr_tests/test_pyspark_step_launcher.py

python_modules/libraries/dagster-aws/setup.py

create dagster-aws-pyspark library with emr_pyspark_step_launcherClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 24432

.buildkite/pipeline.py

Makefile

docs-requirements.txt

docs/next/src/pages/overview/packages/libraries.mdx

docs/sections/api/apidocs/libraries/dagster_aws.rst

docs/sections/api/apidocs/libraries/dagster_aws_pyspark.rst

examples/airline_demo/airline_demo/pipelines.py

examples/airline_demo/setup.py

examples/airline_demo/tox.ini

examples/emr_pyspark/repo.py

examples/emr_pyspark/requirements.txt

examples/emr_pyspark/tox.ini

examples/legacy_examples/Dockerfile

examples/legacy_examples/dagster_examples/simple_pyspark/pipelines.py

examples/legacy_examples/setup.py

examples/legacy_examples/tox.ini

python_modules/libraries/dagster-aws-pyspark/.coveragerc

python_modules/libraries/dagster-aws-pyspark/README.md

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark/__init__.py

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark/pyspark_step_launcher.py

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark/version.py

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark_tests/__init__.py

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark_tests/test_pyspark_step_launcher.py

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark_tests/test_version.py

python_modules/libraries/dagster-aws-pyspark/dev-requirements.txt

python_modules/libraries/dagster-aws-pyspark/requirements.txt

python_modules/libraries/dagster-aws-pyspark/setup.py

python_modules/libraries/dagster-aws-pyspark/tox.ini

python_modules/libraries/dagster-aws/dagster_aws/emr/__init__.py

python_modules/libraries/dagster-aws/dagster_aws/emr/pyspark_step_launcher.py

python_modules/libraries/dagster-aws/dagster_aws_tests/emr_tests/test_pyspark.py

python_modules/libraries/dagster-aws/dagster_aws_tests/emr_tests/test_pyspark_step_launcher.py

python_modules/libraries/dagster-aws/setup.py

create dagster-aws-pyspark library with emr_pyspark_step_launcher
ClosedPublic
Actions

Revision Contents
Changeset List

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark/init.py

python_modules/libraries/dagster-aws-pyspark/dagster_aws_pyspark_tests/init.py

python_modules/libraries/dagster-aws/dagster_aws/emr/init.py