Page MenuHomePhabricator

[EMR 2/N] Refresh Spark solid
ClosedPublic

Authored by nate on Dec 1 2019, 6:27 AM.

Details

Summary

Pulling in part of D1330 as a (mostly) non-breaking change - will pull in the EMR-related parts of D1330 in the next diff

Test Plan

unit

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

nate created this revision.Dec 1 2019, 6:27 AM
nate edited the summary of this revision. (Show Details)Dec 1 2019, 6:30 AM
nate updated this revision to Diff 7029.Dec 1 2019, 6:46 AM

fix snapshot

nate edited the summary of this revision. (Show Details)Dec 1 2019, 6:52 AM
nate added a reviewer: alangenfeld.
alangenfeld accepted this revision.Dec 2 2019, 10:05 PM

if you do want to land any minor breaking changes - make sure to note them in changes.md so we dont lose track

python_modules/libraries/dagster-spark/dagster_spark/solids.py
88–112

what are these changes for - this seems unrelated?

This revision is now accepted and ready to land.Dec 2 2019, 10:05 PM
nate updated this revision to Diff 7547.Dec 11 2019, 12:01 AM

rebase

nate added inline comments.Dec 11 2019, 12:21 AM
python_modules/libraries/dagster-spark/dagster_spark/solids.py
88–112

yeah, just to preserve the ability to share the same spark config across the new resource and the previous spark solid - given that we're starting to move towards breaking changes, I'm going to land this diff and then rip all this out in a follow-up

This revision was automatically updated to reflect the committed changes.