Change Details

I am exploring this now as I want to potentially make this happen in the 0.11.0 timeframe, but that might require deprecations in the 0.10.0 timeframe. The idea here is that we move all artifacts attached to mode to be resources. The way we would explain modes is that they are a set of resource definitions. We could even rename mode to resource set. The invariant in the system is that the entire purpose of a mode to provide a set of pluggable resources, and the the purpose of a resource is to be a thing that can very its implmentation based on mode. Any future abstractions or concepts that vary by mode will be resources. Full stop. There are weird inconsistencies between resources, executors, storages, etc atm. Examples: 1) Only resource definitions can create their resources with a context manager allowing for cleanup. Other resource-ish things do not support that 2) There are different argument orders to ctors 3) Executors, storages etc have embedded names and resources do not. 4) Only intermediate storage supports dependencies on other resources. This should be a generalized capability. I also believe this final merge will cleanup the context initialization process considerably. In order to pull this off we will have to formalize the notion that some resources are "special". For example an executor - Is automatically available to the entire computation and to the system - There can only be one of them - Note: to support the current default behavior where you can switch between multiprocess and in-process out-of-the-box with just a config change, we would need to have an executor definition that makes a decision about which executor to issue, which would require changes in this proposal. In particular the logic around `requires_multiprocess_safe_env` I think it will be interesting to *completely* eliminate ExecutorDefintion entirely and have all the special behavior encoded in the state of the ResourceDefinition itself, but we can defer that discussion as it doesn't change the essence of what is going on here. So what does this do diff do precisely? 1) ExecutorDefinition now inherits from ResourceDefinition 2) InitExecutorContext is gone. InitResourceContext has been expanded to cover the properties that InitExecutorContext has 3) If you pass in a resource definition that is an executor definition, this "takes over" and eliminates the `execution` section of the config. Mode def constructor ensures that the user does not pass in more than one. 4) Moves the check around where an environment is multiprocess-safe into the core. I believe that this should be a generalized feature of resources, but this diff does not do that. I think this is clear code quality and modularity win and I think we should do that change standlone.

TODOI am exploring this now as I want to potentially make this happen in the 0.11.0 timeframe, but that might require deprecations in the 0.10.0 timeframe. The idea here is that we move all artifacts attached to mode to be resources. The way we would explain modes is that they are a set of resource definitions. We could even rename mode to resource set. The invariant in the system is that the entire purpose of a mode to provide a set of pluggable resources, and the the purpose of a resource is to be a thing that can very its implmentation based on mode. Any future abstractions or concepts that vary by mode will be resources. Full stop. There are weird inconsistencies between resources, executors, storages, etc atm. Examples: 1) Only resource definitions can create their resources with a context manager allowing for cleanup. Other resource-ish things do not support that 2) There are different argument orders to ctors 3) Executors, storages etc have embedded names and resources do not. 4) Only intermediate storage supports dependencies on other resources. This should be a generalized capability. I also believe this final merge will cleanup the context initialization process considerably. In order to pull this off we will have to formalize the notion that some resources are "special". For example an executor - Is automatically available to the entire computation and to the system - There can only be one of them - Note: to support the current default behavior where you can switch between multiprocess and in-process out-of-the-box with just a config change, we would need to have an executor definition that makes a decision about which executor to issue, which would require changes in this proposal. In particular the logic around `requires_multiprocess_safe_env` I think it will be interesting to *completely* eliminate ExecutorDefintion entirely and have all the special behavior encoded in the state of the ResourceDefinition itself, but we can defer that discussion as it doesn't change the essence of what is going on here. So what does this do diff do precisely? 1) ExecutorDefinition now inherits from ResourceDefinition 2) InitExecutorContext is gone. InitResourceContext has been expanded to cover the properties that InitExecutorContext has 3) If you pass in a resource definition that is an executor definition, this "takes over" and eliminates the `execution` section of the config. Mode def constructor ensures that the user does not pass in more than one. 4) Moves the check around where an environment is multiprocess-safe into the core. I believe that this should be a generalized feature of resources, but this diff does not do that. I think this is clear code quality and modularity win and I think we should do that change standlone.