If you look in test_io_manager_composites.py, only the IOManagers on the leaf solids take effect. So my expectation was that for root input managers, we would ban them on composite solids for consistency. Are there reasons that doesn't make sense?
we would ban them on composite solids for consistency
looks like we don't hard error on io_manager_key being set on composite, should we? Then also error on root_input_manager on composites?
I think the only place that makes sense to set these keys is on the solid, I'm having a hard time making sense of an other scheme.
I think the ideal behavior is to ban io_manager_key and root_input_manager on @composite_solid but allow these to be set on the inner @solids (which is quite the opposite of this diff). reasons are:
- users can still specify io managers for solids inside composite - imo composite is a way to better organizing solids but limit existing functionalities of its inner solids.
- we can keep solids' reusability - a solid that requires custom io managers can still be included in a composite.
- the dagster machinery can still treat composite as a container - meaning a composite has nothing to do with how io works and it would just defer to its inner solids' set up.
however, to enable inner solids to use custom io managers (which usually take custom inputs/outputs config), here's what's blocking us to do so:
currently, a user can't supply "inputs" and "outputs" for a inner solid in run config, such as:
solids: my_composite_solid: solids: inner_solid: inputs: <input value or config> # composite doesn't take "inputs" or "outputs" config field
if we agree on this plan, i'll go ahead and make composites take "inputs" and "outputs" config for the inner solids.
|13–27 ↗||(On Diff #39492)|
I expected CompositeSolidDefinition - you would have to poke at the definitions via the mappings there but technically its the better place to put it since theoretically someone can construct these via that layer