passing tags looks good to me

thank you for cleaning it up!

I've followed this example flow locally and replaced the token string in the yaml with my own one. It feels pretty lightweight and easy to follow.

would prefer to add an extra staticmethod in ResourceDefinition

@staticmethod
def mock_resource(description=None):
    return ResourceDefinition.hardcoded_resource(value=mock.MagicMock(), description=description)

i believe this duplicates ResourceDefinition.hardcoded_resource and ResourceDefinition.none_resource

mock resource

thank you 🙏 @schrockn

up

bk was flaky. it's passed now

dead links :)

this is AMAZING! thank you!!

• rename reexecute_pipeline(step_keys_to_execute) to `reexecute_pipeline(step_selection)
• persist both step_selection and step_keys_to_execute

rebase master

up

up

up

up

up

enforce that event_list_hook should return a HookExecutionResult

pipeline_def.with_hooks({}) will add hooks to all its solid instance copies and create a copy of the pipeline def

up

solid.hook({}) -> solid.with_hooks({})

HookExecutionResult

what about pipeline_def.with_hooks() to make the pipeline-level step hooks a pipeline property, and invoke both pipeline_def.hook_defs and individual solid.hook_defs in inner_plan_execution_iterator?

+ hook can also decorate pipeline def

up

up

update docstring

rebase

• event_stream -> event_list
• HOOK_SUCCESS -> HOOK_COMPLETED, HOOK_FAILURE -> HOOK_ERRORED
• new HOOK_SKIPPED event when user-defined fn is not triggered

per offline conversation, we are going to defer the decision of whether to provide event stream to a generic hook.

• because we currently don't have a good tradeoff assessment on introducing complexity in a critical code path (the iterator manipulation in execute_plan) VS limiting hooks to wait until the step finishes and get the full list of step events

add more tests for event order

two things to note:
[1] AFAIK event_list can't get the hook events and step events in their actual order. for example, in this test case, the actual order (events in log and console) is

DagsterEventType.STEP_START,
DagsterEventType.STEP_MATERIALIZATION,
DagsterEventType.STEP_MATERIALIZATION,
DagsterEventType.STEP_OUTPUT,
# different order
DagsterEventType.HOOK_SUCCESS,
DagsterEventType.STEP_SUCCESS

but the result.event_list which is generated from inner_plan_execution_iterator will have the order as

DagsterEventType.STEP_START,
# different order
DagsterEventType.HOOK_SUCCESS,
DagsterEventType.STEP_MATERIALIZATION,
DagsterEventType.STEP_MATERIALIZATION,
DagsterEventType.STEP_OUTPUT,
DagsterEventType.STEP_SUCCESS

up

up

revert to Diff 10 -- D19792 should be in next diff

up

up

rename solid_hook -> hook

• implement success/failure hook on top of @solid_hook
• better test case for streaming behavior

streaming behavior in execute_plan

rebase master

rebase master

rebase master

gcp tests failing on master

a generic hook that prints out each event type as the event comes

make _trigger_solid_hook an iterator
move success/failure logic to decorator

up

event_stream