- User Since
- Apr 3 2020, 4:04 PM (8 w, 2 d)
Fri, May 29
I originally started implementing a more general case - something like the partially_configured_resource described above. My impression though was that producing a curried config spec would require working pretty deeply inside the config machinery. I was grepping field snaps and EvaluationStacks, but then stepped back and thought it might not be a good use of my time if my immediate goal was fully configured resources. Maybe I was overcomplicating though and deep surgery isn't required.
I am a data infra eng. I am responsible for how spark contexts are configured prod, stagings test, etc.. However I still want my ops people and devs to be able to control a limited subset of the configuration. E.g. maybe I just want to expose a memory limit and nothing else (contrived example).
Do we think its worth having a static and dynamic API?
Thu, May 28
- from_files instead of from_pkg_resources
Wed, May 27
@alangenfeld you requested changes on an earlier version - do you still have any reservations?
I am a tox / buildkite n00b, so can't claim to have combed through this in great detail, but it seems like the right direction.
Thanks for doing god's work.
remove unrelated changes and fix tests
Tue, May 26
@schrockn thanks for all the feedback. All reasonable. Regarding your main questions - it looks like they're about the SolidAsset, which I threw in on Friday and is a bit half-baked. It could make most sense to leave that for a separate revision?
Fri, May 22
nick and alex naming feedback
Thu, May 21
Wed, May 20
- fix test_emr
- missing yield
@schrockn I tried the single threaded way, per your suggestion, and I think it ended up way cleaner.
- Single thread
Tue, May 19
Mon, May 18
- use real events
Thu, May 14
Wed, May 13
Tue, May 12
Mon, May 11
Fri, May 8
Thu, May 7
Wed, May 6
- Fix common prefix error in to_module_name_based_handle
- Get airline tests working
Tue, May 5
- Get airline tests working
- fix EMR tests and make PySpark resource non-lazy
- remove prints from intermediates_manager
- fix airline tests
- update example snapshots
- pyspark test_resources is no longer applicable
- fix airline_demo prod_base.yaml
- get simple_pyspark example working and fix handle tests
- remove @pyspark_solid
Make test pass intermediates, clean up airline demo, enforce step launcher config invariant
Mon, May 4
rsync before test_do_it_live_emr
Take out step_launcher_resource_key from solid definition
Clean up loose ends and get test_do_it_live_emr working
Fri, May 1
- Doc LocalExternalStepLauncher and internal_step_launcher -> no_step_launcher
Apr 30 2020
Find launcher using inheritance rather than resource key name
Apr 29 2020
I think we should start with the outer retries since we do want to be able retry the launch - we can add the inner retries later
Retries and check types in new
Replace InternalStepLauncher with None