I am exploring this now as I want to potentially make this
happen in the 0.11.0 timeframe, but that might require deprecations in
the 0.10.0 timeframe.
The idea here is that we move all artifacts attached to mode to be
resources. The way we would explain modes is that they are a set of
resource definitions. We could even rename mode to resource set.
The invariant in the system is that the entire purpose of a mode to
provide a set of pluggable resources, and the the purpose of a resource
is to be a thing that can very its implmentation based on mode. Any
future abstractions or concepts that vary by mode will be resources.
There are weird inconsistencies between resources, executors,
storages, etc atm. Examples:
- Only resource definitions can create their resources with a context manager allowing for cleanup. Other resource-ish things do not support that
- There are different argument orders to ctors
- Executors, storages etc have embedded names and resources do not.
- Only intermediate storage supports dependencies on other resources. This should be a generalized capability.
I also believe this final merge will cleanup the context initialization
In order to pull this off we will have to formalize the notion that some
resources are "special". For example an executor
- Is automatically available to the entire computation and to the system
- There can only be one of them
- Note: to support the current default behavior where you can switch
between multiprocess and in-process out-of-the-box with just a config
change, we would need to have an executor definition that makes a
decision about which executor to issue, which would require changes in
this proposal. In particular the logic around
I think it will be interesting to *completely* eliminate
ExecutorDefintion entirely and have all the special behavior encoded
in the state of the ResourceDefinition itself, but we can defer that
discussion as it doesn't change the essence of what is going on here.
So what does this do diff do precisely?
- ExecutorDefinition now inherits from ResourceDefinition
- InitExecutorContext is gone. InitResourceContext has been expanded to cover the properties that InitExecutorContext has
- If you pass in a resource definition that is an executor definition, this "takes over" and eliminates the execution section of the config. Mode def constructor ensures that the user does not pass in more than one.
- Moves the check around where an environment is multiprocess-safe into the core. I believe that this should be a generalized feature of resources, but this diff does not do that. I think this is clear code quality and modularity win and I think we should do that change standlone.