This diff aims to solve the naming issue that was introduced by the configured API. It also adds configured to API docs.
Naming Issue
The configured API performs a Definition -> Definition transformation. There are various definition types where configurable is useful, hence having a single interface between each, but there is variation between the required parameters of different definition types. This creates awkward API ergonomics under certain scenarios. This is most acute in the case of the name parameter. Consider that SolidDefinitions must have globally unique names, while ResourceDefinition has no name parameter whatsoever.
Possible Solutions
Current Solution
The current solution is that name exists as an optional parameter on the configured function, which will cause an error if not provided to a SolidDefinition, and cause an error if provided to a ResourceDefinition.
CallableNode Solution
One possible solution is to change the configured API to emit a CallableNode instead of a SolidDefinition, similar to how the alias, tag, and with_hook functions work (for the solid case). CallableNode(s) can be auto-aliased, avoiding the need for the name parameter.
However, CallableNodes are pipeline scoped, meaning they can only be declared within the context of a pipeline (can't be reused between pipelines). This would create a discrepancy between how configured is used on solids, vs how it is used on other definition types.
I think putting configured on CallableNode is also a conceptual mismatch. configured is a form of currying. By transforming the set of inputs (considering config schema as an input), configured is fundamentally changing the solid, whereas fxns like alias and tag are changing metadata surrounding the solid.
In Dagit, we can see this dichotomy as a solid "invocation" vs a solid definition. CallableNodes are linked back to the original definition when invoked.
Overall, it seems that solving the problem in this way leads to:
- constraining the scenarios in which one can use the API
- Conceptual thrash (we don't treat CallableNodes as distinct entities, while viewing things from a currying perspective suggests we should)
- Represented weirdly in Dagit
Split interface solution
The other solution (and the one I will be advocating for) would be to simply have different interfaces for different types of definitions, to mirror the parameters that they require.
To illustrate this, we would have an API NamedConfigurableDefinition interface which has a positional arg for name on the configured function, and an AnonymousConfigurableDefinition interface which has no name arg.
This trades off API parity between configured on different definition types, for clarity about required parameters. While this could potentially create confusion for someone using configured on two different APIs, I think this is a much more momentary confusion than the confusing cases in each alternative.
Tradeoffs
Confusing case in new interfaces approach: hm I used solid.configured and it wanted a name, and I used resource.configured and it didn't want a name.
Confusing case in CallableNode approach: hm I tried to use solid.configured outside of pipeline scope and it didn't work
Confusing case in current approach: hm I tried to use solid.configured, name was a positional arg so I didn't provide it, but then I received an error. Why is name a positional arg if it is required? || hm I tried to use resource.configured, name was a positional arg so I provided it, but then I received an error. Why is name a positional arg if it isn't usable?
In my opinion, surmounting the confusing case in the "new interfaces approach" will be a lot easier for a user, and also a lot easier for us to communicate (we can add a note to the documentation saying "hey solids have this constraint of being uniquely named so we have to provide a name parameter", and that is (imo) less conceptual overhead than communicating in the other scenarios.
From an internal perspective, I think having different interfaces for different definitions is an acknowledgement of this uniqueness constraint on solids vs other types of definitions, and is better than workarounds to pretend it isn't there.
Other potential weirdness
The configured decorator. Name is still a positional arg on this, which is probably fine because we just steal the function name, but it is a bit confusing.
Since providing a string is potentially valid config, mismatch between name and config is possible.
