Page MenuHomePhabricator

yuhan (Yuhan Luo)
Logistics

Projects

User does not belong to any projects.

User Details

User Since
Feb 28 2020, 11:37 PM (23 w, 3 d)

Recent Activity

Today

yuhan added inline comments to D4152: Pass tags to PipelineSubsetDef.
Tue, Aug 11, 9:13 PM
yuhan accepted D4152: Pass tags to PipelineSubsetDef.

passing tags looks good to me

Tue, Aug 11, 8:29 PM
yuhan accepted D4153: remove deprecated env param from cli pipeline commands.

thank you for cleaning it up!

Tue, Aug 11, 7:21 PM

Yesterday

yuhan planned changes to D4107: step-selection-1 parse step selection.
Mon, Aug 10, 9:26 PM

Sat, Aug 8

yuhan added a comment to D4119: [docs] examples/hooks.

I've followed this example flow locally and replaced the token string in the yaml with my own one. It feels pretty lightweight and easy to follow.

Sat, Aug 8, 12:22 AM

Fri, Aug 7

yuhan requested review of D4147: ResourceDefinition.mock_resource.
Fri, Aug 7, 11:41 PM

Thu, Aug 6

yuhan published D4131: migration guide only includes major releases for review.
Thu, Aug 6, 9:42 PM
yuhan committed R1:be52c1ce07db: migration guide (authored by sandyryza).
migration guide
Thu, Aug 6, 9:32 PM
yuhan closed D4036: migration guide.
Thu, Aug 6, 9:32 PM
yuhan commandeered D4036: migration guide.
Thu, Aug 6, 9:31 PM
yuhan added a comment to D4121: Just for looking: This is an example of some helpers we could use to help mock resources.

would prefer to add an extra staticmethod in ResourceDefinition

@staticmethod
def mock_resource(description=None):
    return ResourceDefinition.hardcoded_resource(value=mock.MagicMock(), description=description)
Thu, Aug 6, 9:13 PM
yuhan committed R1:0b3dcaf36666: [docs] examples/hooks (authored by yuhan).
[docs] examples/hooks
Thu, Aug 6, 9:08 PM
yuhan closed D4119: [docs] examples/hooks.
Thu, Aug 6, 9:08 PM
yuhan added inline comments to D4119: [docs] examples/hooks.
Thu, Aug 6, 9:07 PM
yuhan abandoned D4121: Just for looking: This is an example of some helpers we could use to help mock resources.

i believe this duplicates ResourceDefinition.hardcoded_resource and ResourceDefinition.none_resource

Thu, Aug 6, 8:14 PM
yuhan committed R1:35a04a55a29b: airline_demo readme (authored by yuhan).
airline_demo readme
Thu, Aug 6, 8:11 PM
yuhan closed D4116: airline_demo readme.
Thu, Aug 6, 8:11 PM
yuhan updated the test plan for D4119: [docs] examples/hooks.
Thu, Aug 6, 8:11 PM
yuhan updated the diff for D4119: [docs] examples/hooks.

mock resource

Thu, Aug 6, 8:10 PM
yuhan added inline comments to D4119: [docs] examples/hooks.
Thu, Aug 6, 8:09 PM
yuhan added inline comments to D4119: [docs] examples/hooks.
Thu, Aug 6, 5:03 PM
yuhan commandeered D4121: Just for looking: This is an example of some helpers we could use to help mock resources.

thank you 🙏 @schrockn

Thu, Aug 6, 4:47 PM
yuhan updated the diff for D4116: airline_demo readme.

up

Thu, Aug 6, 4:46 PM
yuhan requested review of D4119: [docs] examples/hooks.
Thu, Aug 6, 5:59 AM
yuhan committed R1:440f54899d1c: fix airflow_ingest example import (authored by yuhan).
fix airflow_ingest example import
Thu, Aug 6, 5:05 AM
yuhan closed D4117: fix airflow_ingest example import.
Thu, Aug 6, 5:05 AM
yuhan added a comment to D4116: airline_demo readme.

bk was flaky. it's passed now

Thu, Aug 6, 3:51 AM
yuhan requested review of D4117: fix airflow_ingest example import.
Thu, Aug 6, 3:30 AM
yuhan updated the diff for D4116: airline_demo readme.

dead links :)

Thu, Aug 6, 3:21 AM
yuhan published D4116: airline_demo readme for review.
Thu, Aug 6, 12:58 AM

Wed, Aug 5

yuhan added a comment to D4113: [docs] dagit overview.

this is AMAZING! thank you!!

Wed, Aug 5, 9:56 PM
yuhan updated the summary of D4107: step-selection-1 parse step selection.
Wed, Aug 5, 9:15 PM
yuhan updated the diff for D4107: step-selection-1 parse step selection.
  • rename reexecute_pipeline(step_keys_to_execute) to `reexecute_pipeline(step_selection)
  • persist both step_selection and step_keys_to_execute
Wed, Aug 5, 9:14 PM
yuhan committed R1:9769e51eaa5f: solid-hook-3 hooks can decorate pipeline defs (authored by yuhan).
solid-hook-3 hooks can decorate pipeline defs
Wed, Aug 5, 7:40 PM
yuhan closed D4014: solid-hook-3 hooks can decorate pipeline defs.
Wed, Aug 5, 7:40 PM
yuhan updated the diff for D4014: solid-hook-3 hooks can decorate pipeline defs.

rebase master

Wed, Aug 5, 7:19 PM
yuhan updated the diff for D4014: solid-hook-3 hooks can decorate pipeline defs.

up

Wed, Aug 5, 6:52 PM
yuhan committed R1:6d0263f4f731: solid-hook-2 apply hooks on solid instances (authored by yuhan).
solid-hook-2 apply hooks on solid instances
Wed, Aug 5, 6:37 PM
yuhan closed D3979: solid-hook-2 apply hooks on solid instances.
Wed, Aug 5, 6:37 PM
yuhan closed D3977: solid-hook-1 @hook and HookDefinition.
Wed, Aug 5, 6:34 PM
yuhan committed R1:5cb9648f32ed: solid-hook-1 @hook and HookDefinition (authored by yuhan).
solid-hook-1 @hook and HookDefinition
Wed, Aug 5, 6:34 PM
yuhan added inline comments to D4107: step-selection-1 parse step selection.
Wed, Aug 5, 6:31 PM
yuhan updated the summary of D4107: step-selection-1 parse step selection.
Wed, Aug 5, 6:29 PM
yuhan requested review of D4107: step-selection-1 parse step selection.
Wed, Aug 5, 6:24 PM
yuhan committed R1:95749f8327d2: fix empty set/frozenset serde (authored by yuhan).
fix empty set/frozenset serde
Wed, Aug 5, 4:53 PM
yuhan closed D4105: fix empty set/frozenset serde.
Wed, Aug 5, 4:53 PM
yuhan requested review of D4105: fix empty set/frozenset serde.
Wed, Aug 5, 4:40 PM
yuhan retitled D4014: solid-hook-3 hooks can decorate pipeline defs from solid-hook-3 hooks can decorate solid defs and pipeline defs to solid-hook-3 hooks can decorate pipeline defs.
Wed, Aug 5, 4:35 PM
yuhan updated the diff for D3979: solid-hook-2 apply hooks on solid instances.

up

Wed, Aug 5, 4:34 PM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

up

Wed, Aug 5, 4:32 PM
yuhan accepted D4101: fix compute log manager config docs.
Wed, Aug 5, 3:48 AM

Tue, Aug 4

yuhan updated the diff for D4014: solid-hook-3 hooks can decorate pipeline defs.

up

Tue, Aug 4, 10:50 PM
yuhan updated the diff for D3979: solid-hook-2 apply hooks on solid instances.

up

Tue, Aug 4, 10:48 PM
yuhan added inline comments to D3977: solid-hook-1 @hook and HookDefinition.
Tue, Aug 4, 10:48 PM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

enforce that event_list_hook should return a HookExecutionResult

Tue, Aug 4, 10:46 PM
yuhan updated the diff for D4014: solid-hook-3 hooks can decorate pipeline defs.

pipeline_def.with_hooks({}) will add hooks to all its solid instance copies and create a copy of the pipeline def

Tue, Aug 4, 10:36 PM
yuhan updated the diff for D3979: solid-hook-2 apply hooks on solid instances.

up

Tue, Aug 4, 9:18 PM
yuhan added inline comments to D3979: solid-hook-2 apply hooks on solid instances.
Tue, Aug 4, 9:11 PM
yuhan updated the diff for D3979: solid-hook-2 apply hooks on solid instances.

solid.hook({}) -> solid.with_hooks({})

Tue, Aug 4, 7:24 PM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

HookExecutionResult

Tue, Aug 4, 7:14 PM
yuhan added a comment to D4014: solid-hook-3 hooks can decorate pipeline defs.

what about pipeline_def.with_hooks() to make the pipeline-level step hooks a pipeline property, and invoke both pipeline_def.hook_defs and individual solid.hook_defs in inner_plan_execution_iterator?

Tue, Aug 4, 5:24 PM

Mon, Aug 3

yuhan updated the summary of D4014: solid-hook-3 hooks can decorate pipeline defs.
Mon, Aug 3, 11:00 PM
yuhan retitled D4014: solid-hook-3 hooks can decorate pipeline defs from solid-hook-3 hooks can decorate solid defs to solid-hook-3 hooks can decorate solid defs and pipeline defs.
Mon, Aug 3, 10:59 PM
yuhan updated the diff for D4014: solid-hook-3 hooks can decorate pipeline defs.

+ hook can also decorate pipeline def

Mon, Aug 3, 10:56 PM
yuhan updated the summary of D3979: solid-hook-2 apply hooks on solid instances.
Mon, Aug 3, 9:53 PM
yuhan updated the diff for D3979: solid-hook-2 apply hooks on solid instances.

up

Mon, Aug 3, 9:22 PM
yuhan updated the diff for D3979: solid-hook-2 apply hooks on solid instances.

up

Mon, Aug 3, 9:14 PM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

update docstring

Mon, Aug 3, 8:18 PM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

rebase

Mon, Aug 3, 6:06 PM
yuhan updated the summary of D3977: solid-hook-1 @hook and HookDefinition.
Mon, Aug 3, 5:41 PM

Sat, Aug 1

yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.
  • event_stream -> event_list
  • HOOK_SUCCESS -> HOOK_COMPLETED, HOOK_FAILURE -> HOOK_ERRORED
  • new HOOK_SKIPPED event when user-defined fn is not triggered
Sat, Aug 1, 1:02 AM

Fri, Jul 31

yuhan planned changes to D3977: solid-hook-1 @hook and HookDefinition.

per offline conversation, we are going to defer the decision of whether to provide event stream to a generic hook.

  • because we currently don't have a good tradeoff assessment on introducing complexity in a critical code path (the iterator manipulation in execute_plan) VS limiting hooks to wait until the step finishes and get the full list of step events
Fri, Jul 31, 10:31 PM
yuhan accepted D4057: delete toys out of legacy_examples.

byebye

Fri, Jul 31, 8:59 PM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

add more tests for event order

Fri, Jul 31, 12:58 AM

Thu, Jul 30

yuhan added a comment to D3977: solid-hook-1 @hook and HookDefinition.

two things to note:
[1] AFAIK event_list can't get the hook events and step events in their actual order. for example, in this test case, the actual order (events in log and console) is

DagsterEventType.STEP_START,
DagsterEventType.STEP_MATERIALIZATION,
DagsterEventType.STEP_MATERIALIZATION,
DagsterEventType.STEP_OUTPUT,
# different order
DagsterEventType.HOOK_SUCCESS,
DagsterEventType.STEP_SUCCESS

but the result.event_list which is generated from inner_plan_execution_iterator will have the order as

DagsterEventType.STEP_START,
# different order
DagsterEventType.HOOK_SUCCESS,
DagsterEventType.STEP_MATERIALIZATION,
DagsterEventType.STEP_MATERIALIZATION,
DagsterEventType.STEP_OUTPUT,
DagsterEventType.STEP_SUCCESS
Thu, Jul 30, 11:16 PM
yuhan added inline comments to D3977: solid-hook-1 @hook and HookDefinition.
Thu, Jul 30, 6:58 PM
yuhan updated the diff for D3979: solid-hook-2 apply hooks on solid instances.

up

Thu, Jul 30, 5:50 AM
yuhan updated the diff for D3979: solid-hook-2 apply hooks on solid instances.

up

Thu, Jul 30, 5:42 AM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

revert to Diff 10 -- D19792 should be in next diff

Thu, Jul 30, 5:37 AM
yuhan updated the diff for D4014: solid-hook-3 hooks can decorate pipeline defs.

up

Thu, Jul 30, 5:31 AM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

up

Thu, Jul 30, 5:28 AM
yuhan retitled D3977: solid-hook-1 @hook and HookDefinition from solid-hook-1 @solid_hook and HookDefinition to solid-hook-1 @hook and HookDefinition.
Thu, Jul 30, 5:24 AM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

rename solid_hook -> hook

Thu, Jul 30, 5:24 AM
yuhan committed R1:1771b49f58c6: docs_snippet test fix - skip download file in tests (authored by yuhan).
docs_snippet test fix - skip download file in tests
Thu, Jul 30, 4:53 AM
yuhan closed D4044: docs_snippet test fix - skip download file in tests.
Thu, Jul 30, 4:53 AM
yuhan published D4044: docs_snippet test fix - skip download file in tests for review.
Thu, Jul 30, 4:41 AM
yuhan added inline comments to D3977: solid-hook-1 @hook and HookDefinition.
Thu, Jul 30, 1:03 AM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.
  • implement success/failure hook on top of @solid_hook
  • better test case for streaming behavior
Thu, Jul 30, 12:57 AM

Wed, Jul 29

yuhan added inline comments to D3977: solid-hook-1 @hook and HookDefinition.
Wed, Jul 29, 8:14 PM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

streaming behavior in execute_plan

Wed, Jul 29, 8:13 PM

Mon, Jul 27

yuhan updated the diff for D4014: solid-hook-3 hooks can decorate pipeline defs.

rebase master

Mon, Jul 27, 6:31 PM
yuhan updated the diff for D3979: solid-hook-2 apply hooks on solid instances.

rebase master

Mon, Jul 27, 6:30 PM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

rebase master

Mon, Jul 27, 6:29 PM
yuhan published D4014: solid-hook-3 hooks can decorate pipeline defs for review.

gcp tests failing on master

Mon, Jul 27, 5:32 PM

Sat, Jul 25

yuhan added a comment to D3977: solid-hook-1 @hook and HookDefinition.

a generic hook that prints out each event type as the event comes

Sat, Jul 25, 1:18 AM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

make _trigger_solid_hook an iterator
move success/failure logic to decorator

Sat, Jul 25, 1:17 AM

Fri, Jul 24

yuhan committed R1:68ee2fbc777d: release automation check_versions bug (authored by yuhan).
release automation check_versions bug
Fri, Jul 24, 6:20 PM
yuhan closed D3999: release automation check_versions bug.
Fri, Jul 24, 6:20 PM
yuhan updated the diff for D3979: solid-hook-2 apply hooks on solid instances.

up

Fri, Jul 24, 5:10 PM
yuhan updated the diff for D3977: solid-hook-1 @hook and HookDefinition.

event_stream

Fri, Jul 24, 5:09 PM