Just to give you guys a sense of what this will look like. It is truly unfortunate. However SolidInvocation is exported top-level so we will need a story.
The more obvious structure:
class NodeInvocation(namedtuple("Solid", "name alias tags hook_defs")):
def __new__(cls, name, alias=None, tags=None, hook_defs=None):
name = check.str_param(name, "name")
alias = check.opt_str_param(alias, "alias")
tags = frozentags(check.opt_dict_param(tags, "tags", value_type=str, key_type=str))
hook_defs = frozenset(check.opt_set_param(hook_defs, "hook_defs", of_type=HookDefinition))
return super(cls, NodeInvocation).__new__(cls, name, alias, tags, hook_defs)
class SolidInvocation(NodeInvocation):
"""Identifies an instance of a solid in a pipeline dependency structure.
Args:
name (str): Name of the solid of which this is an instance.
alias (Optional[str]): Name specific to this instance of the solid. Necessary when there are
multiple instances of the same solid.
tags (Optional[Dict[str, Any]]): Optional tags values to extend or override those
set on the solid definition.
hook_defs (Optional[Set[HookDefinition]]): A set of hook definitions applied to the
solid instance.
Examples:
.. code-block:: python
pipeline = PipelineDefinition(
solid_defs=[solid_1, solid_2]
dependencies={
SolidInvocation('solid_1', alias='other_name') : {
'input_name' : DependencyDefinition('solid_1'),
},
'solid_2' : {
'input_name': DependencyDefinition('other_name'),
},
}
)
In general, users should prefer not to construct this class directly or use the
:py:class:`PipelineDefinition` API that requires instances of this class. Instead, use the
:py:func:`@pipeline <pipeline>` API:
.. code-block:: python
@pipeline
def pipeline():
other_name = solid_1.alias('other_name')
solid_2(other_name(solid_1))
"""
def __new__(cls, name, alias=None, tags=None, hook_defs=None):
name = check.str_param(name, "name")
alias = check.opt_str_param(alias, "alias")
tags = frozentags(check.opt_dict_param(tags, "tags", value_type=str, key_type=str))
hook_defs = frozenset(check.opt_set_param(hook_defs, "hook_defs", of_type=HookDefinition))
return super(cls, SolidInvocation).__new__(cls, name, alias, tags, hook_defsResults in the unfortunate error:
python_modules/dagster/dagster/core/definitions/dependency.py:153: in __new__
return super(cls, SolidInvocation).__new__(cls, name, alias, tags, hook_defs)
python_modules/dagster/dagster/core/definitions/dependency.py:112: in __new__
return super(cls, NodeInvocation).__new__(cls, name, alias, tags, hook_defs)
E TypeError: super(type, obj): obj must be an instance or subtype of typeSo the solution I came up was to have an inheritance hierarchy and then have it
contain a namedtuple. I did containment to ensure maintainence precisely the same
hashing and equality semantics. (Side note: hooks seem problematic here)
I'd love to have a better solution here so I'm all ears
Depends on D4912