This scrubs run config from all the lower level execution APIs, only exposing pipeline_run on the attribute.
The top-level execution API functions now contain the following:
- execute_run and execute_run_iterator take a PipelineRun object.
- execute_pipeline and execute_pipeline_iterator take in a RunConfig and construct a PipelineRun that is persisted using the instance run storage
- execute_plan and execute_plan_iterator take in a RunConfig and construct an ephemeral PipelineRun that is not persisted.
In order to make this work with the multiprocessing engine, we have to be able to reconstruct a run config from a pipeline run and use it to set up the environment/context again, using execute_plan_iterator.
We also have a slightly confusing API now with the context mapper. The config traversal context is now different than the execution context in that the config traversal context contains a RunConfig while the execution context contains a PipelineRun. Would love to discuss the context mapper and what the appropriate API should be.