I originally started implementing a more general case - something like the partially_configured_resource described above. My impression though was that producing a curried config spec would require working pretty deeply inside the config machinery. I was grepping field snaps and EvaluationStacks, but then stepped back and thought it might not be a good use of my time if my immediate goal was fully configured resources. Maybe I was overcomplicating though and deep surgery isn't required.

I am a data infra eng. I am responsible for how spark contexts are configured prod, stagings test, etc.. However I still want my ops people and devs to be able to control a limited subset of the configuration. E.g. maybe I just want to expose a memory limit and nothing else (contrived example).

Do we think its worth having a static and dynamic API?

black

• from_files instead of from_pkg_resources

@alangenfeld you requested changes on an earlier version - do you still have any reservations?

fewer globals

black

try to fix 3.5 tests

I am a tox / buildkite n00b, so can't claim to have combed through this in great detail, but it seems like the right direction.

Thanks for doing god's work.

remove unrelated changes and fix tests

• more lakehouse

@schrockn thanks for all the feedback. All reasonable. Regarding your main questions - it looks like they're about the SolidAsset, which I threw in on Friday and is a bit half-baked. It could make most sense to leave that for a separate revision?

nick and alex naming feedback

Fix a bug

• fix test_emr

• missing yield

@schrockn I tried the single threaded way, per your suggestion, and I think it ended up way cleaner.

• Single thread

• use real events

• Fix common prefix error in to_module_name_based_handle

• Get airline tests working

• Get airline tests working

• DagsterSubprocessError

• fix EMR tests and make PySpark resource non-lazy

• remove prints from intermediates_manager

• fix airline tests

• black
• update example snapshots
• pyspark test_resources is no longer applicable

• fix airline_demo prod_base.yaml
• get simple_pyspark example working and fix handle tests

• remove @pyspark_solid

Make test pass intermediates, clean up airline demo, enforce step launcher config invariant

Fix ETH

rsync before test_do_it_live_emr

Take out step_launcher_resource_key from solid definition

Clean up loose ends and get test_do_it_live_emr working

• fix test broken by rebase

Spacing merge issue

• Doc LocalExternalStepLauncher and internal_step_launcher -> no_step_launcher

Find launcher using inheritance rather than resource key name

I think we should start with the outer retries since we do want to be able retry the launch - we can add the inner retries later

Retries and check types in new

Replace InternalStepLauncher with None

Black