Cleaning up / improving the interface for defining Spark jobs.
The motivation here is that I'd like to better deliver on the promise of seamless local development <> prod deployment.
To do that for e.g. local spark <> EMR, I think we need to decouple the solid implementation from the environment configuration required to physically execute the Spark job; i.e. move the latter to a resource as here—otherwise, the user is forced to rewrite their jobs with different solids for production.
Downside is that we end up with one resource def per solid (see the multiple jobs test), but (with a bit more work here) at least you wouldn't have to replace all of your solids in your production version, and you could just swap out the local Spark resources for EMR resources.
Would love ideas on if there's a better way to do this!