Page MenuHomeElementl

create dagster-aws-pyspark library with emr_pyspark_step_launcher
ClosedPublic

Authored by sandyryza on Oct 16 2020, 9:00 PM.

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 16 2020, 9:18 PM
Harbormaster failed remote builds in B19737: Diff 23956!
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 16 2020, 10:06 PM
Harbormaster failed remote builds in B19742: Diff 23961!
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 16 2020, 11:18 PM
Harbormaster failed remote builds in B19744: Diff 23963!
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 4:15 PM
Harbormaster failed remote builds in B19756: Diff 23977!
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 6:11 PM
Harbormaster failed remote builds in B19771: Diff 23991!
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 7:04 PM
Harbormaster failed remote builds in B19778: Diff 23999!
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 8:48 PM
Harbormaster failed remote builds in B19797: Diff 24020!
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 10:15 PM
Harbormaster failed remote builds in B19810: Diff 24034!
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 19 2020, 11:05 PM
Harbormaster failed remote builds in B19815: Diff 24040!
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 20 2020, 12:25 AM
Harbormaster failed remote builds in B19819: Diff 24045!

should we call it dagster-emr?

This revision now requires changes to proceed.Oct 23 2020, 10:45 PM

should we call it dagster-emr?

dagster-aws has some non-PySpark EMR stuff in it (EmrJobRunner). The motivation for this change came from a user who wanted to be able to use that stuff without pulling in pyspark dependencies.

This revision is now accepted and ready to land.Oct 24 2020, 12:46 AM