Page MenuHomeElementl

yuhan (Yuhan Luo)
User

Projects

User does not belong to any projects.

User Details

User Since
Feb 28 2020, 11:37 PM (74 w, 5 d)

Recent Activity

Mon, Aug 2

yuhan accepted D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

my hunch is later in the stack (after D9085), this shouldn't be too large of a change, e.g. could potentially toggle is on io manager config?

Mon, Aug 2, 5:18 PM

Fri, Jul 30

yuhan added a comment to D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

in terms of user experience, i think it's fine to implicitly add a tag to a run - some other features work like that, e.g. we implicitly add parent_run_id and root_run_id as tags to new runs, and solid/step selection too. besides, it's also good that users can view those info in the Runs page such as:

image.png (802×2 px, 185 KB)

Fri, Jul 30, 7:13 PM
yuhan added a comment to D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

Which brings us back to Alex's initial q: what's the right way to toggle memoization on and off?

Fri, Jul 30, 6:23 PM
yuhan commandeered D6145: Use IO managers for Dagstermill serialization.
Fri, Jul 30, 5:37 PM
yuhan planned changes to D8981: add `from_failure` bool arg to rexecute pipeline apis.

trying enum

Fri, Jul 30, 5:12 PM
yuhan published D4698: Prototype: Add test utility for dagstermill solids for review.
Fri, Jul 30, 12:17 AM

Thu, Jul 29

yuhan added a comment to D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

in terms of modeling similar "load-from-other-run-in-the-run-group look-ups" in the same way, i could see a path forward that we consolidate logics into something like [get_retry_steps_from_execution_plan](https://github.com/dagster-io/dagster/blob/bfb90e8b0b442f657e5256082d3116aefa8c330b/python_modules/dagster/dagster/core/execution/plan/resume_retry.py#L31) - at a high level, all the paths are basically fn(historical execution plann, logs) -> execution_plan.

Thu, Jul 29, 7:46 PM
yuhan accepted D9059: [memoization 2/n] provide default base dir for versioned fs io manager.
Thu, Jul 29, 7:26 PM
yuhan accepted D9144: [dagit] Repair repo specificity on Run details and re-execution.
Thu, Jul 29, 6:14 PM

Wed, Jul 28

yuhan closed D9110: dagstermill - yield_result output path account for aliased solids and composite solids.
Wed, Jul 28, 11:23 PM
yuhan committed R1:f1c0a4f53e57: dagstermill - yield_result output path account for aliased solids and composite… (authored by yuhan).
dagstermill - yield_result output path account for aliased solids and composite…
Wed, Jul 28, 11:23 PM
yuhan requested review of D9110: dagstermill - yield_result output path account for aliased solids and composite solids.
Wed, Jul 28, 4:37 AM

Tue, Jul 27

yuhan closed D9071: mypy run storage.
Tue, Jul 27, 11:08 PM
yuhan committed R1:bc9d4942212f: mypy run storage (authored by yuhan).
mypy run storage
Tue, Jul 27, 11:08 PM
yuhan accepted D9103: Add missing imports to hacker_news example repo.py.
Tue, Jul 27, 11:02 PM
yuhan updated the diff for D9071: mypy run storage.

gql

Tue, Jul 27, 10:16 PM
yuhan updated the diff for D9071: mypy run storage.

loosen handle_run_event error
add error handling to in mem storage

Tue, Jul 27, 8:43 PM
yuhan added inline comments to D9059: [memoization 2/n] provide default base dir for versioned fs io manager.
Tue, Jul 27, 6:56 PM
yuhan closed D9017: [docs] sensor concept include run status sensor.
Tue, Jul 27, 6:47 PM
yuhan committed R1:67442bdea2da: [docs] sensor concept include run status sensor (authored by yuhan).
[docs] sensor concept include run status sensor
Tue, Jul 27, 6:47 PM
yuhan added a comment to D9017: [docs] sensor concept include run status sensor.

yup pipeline failure sensor is already nested under the run status sensor and solid hooks are still linked - i didn't update the failure sensor section in this diff.

Tue, Jul 27, 6:46 PM
yuhan updated the diff for D9071: mypy run storage.

DagsterRunNotFoundError

Tue, Jul 27, 6:41 PM
yuhan closed D9053: dagstermill - fix: yield_result temp path includes solid name.
Tue, Jul 27, 5:35 PM
yuhan committed R1:ad5abf66f0b2: dagstermill - fix: yield_result temp path includes solid name (authored by yuhan).
dagstermill - fix: yield_result temp path includes solid name
Tue, Jul 27, 5:35 PM
yuhan requested review of D9071: mypy run storage.
Tue, Jul 27, 2:09 AM

Mon, Jul 26

yuhan requested review of D9053: dagstermill - fix: yield_result temp path includes solid name.
Mon, Jul 26, 6:16 PM

Fri, Jul 23

yuhan accepted D9000: add title and description to graph migration page.
Fri, Jul 23, 11:25 PM
yuhan accepted D9013: Add dbt pipeline to example repo, synchronize with internal.

this is great!

Fri, Jul 23, 11:24 PM
yuhan accepted D9047: Move invalid step logic down into gRPC, to reduce gRPC calls.
Fri, Jul 23, 11:00 PM
yuhan accepted D9035: tailwind config for highlighting back links in source snippets.
Fri, Jul 23, 1:18 AM

Thu, Jul 22

yuhan requested review of D9017: [docs] sensor concept include run status sensor.
Thu, Jul 22, 7:53 PM
yuhan added inline comments to D9021: changes for 0.12.3.
Thu, Jul 22, 7:11 PM
yuhan closed D9015: run_status_sensor api doc.
Thu, Jul 22, 6:26 PM
yuhan committed R1:1b305b5db8d7: run_status_sensor api doc (authored by yuhan).
run_status_sensor api doc
Thu, Jul 22, 6:26 PM
yuhan closed D8900: RunStatusSensorDefinition and @run_status_sensor.
Thu, Jul 22, 5:43 PM
yuhan committed R1:306593aa82c9: RunStatusSensorDefinition and @run_status_sensor (authored by yuhan).
RunStatusSensorDefinition and @run_status_sensor
Thu, Jul 22, 5:43 PM
yuhan closed D8896: consolidate event type to run status lookup.
Thu, Jul 22, 5:29 PM
yuhan committed R1:19f525d6f7be: consolidate event type to run status lookup (authored by yuhan).
consolidate event type to run status lookup
Thu, Jul 22, 5:29 PM
yuhan added inline comments to D8900: RunStatusSensorDefinition and @run_status_sensor.
Thu, Jul 22, 12:35 AM
yuhan updated the diff for D8900: RunStatusSensorDefinition and @run_status_sensor.

PipelineSensorCursor -> RunStatusSensorCursor + backcompat fallback

Thu, Jul 22, 12:35 AM

Wed, Jul 21

yuhan added inline comments to D8900: RunStatusSensorDefinition and @run_status_sensor.
Wed, Jul 21, 11:33 PM
yuhan added a comment to D8981: add `from_failure` bool arg to rexecute pipeline apis.

one alternative would be to extend step_selection to also take an enum entry or marker type for describing a step selection to be calculated by dagster.

we could

  • extend step_selection to take StepSelection.FROM_FAILURE), step_selection=StepSelection.from_selection(["some_solid*"]), and ["some_solid*"] (backcompat)
  • or a more generic arg reexecute_option and deprecate step_selection, it'd take ReexecuteOption.FROM_FAILURE, ReexecuteOption.from_selection(["some_solid*"]))
Wed, Jul 21, 9:23 PM
yuhan requested review of D8981: add `from_failure` bool arg to rexecute pipeline apis.
Wed, Jul 21, 6:54 PM

Tue, Jul 20

yuhan accepted D8980: Fix sensor testing docs.
Tue, Jul 20, 11:38 PM
yuhan accepted D8976: [new-project] sensor targets non-existent mode.

not directly related to this diff: how do people feel about having a separate new-project cli for the crag stuff?

Tue, Jul 20, 11:23 PM
yuhan updated the diff for D8900: RunStatusSensorDefinition and @run_status_sensor.

PipelineSensor -> RunStatusSensor

Tue, Jul 20, 11:18 PM
yuhan added inline comments to D8900: RunStatusSensorDefinition and @run_status_sensor.
Tue, Jul 20, 11:12 PM
yuhan accepted D8974: remove bad use of 'in'.
Tue, Jul 20, 9:03 PM

Mon, Jul 19

yuhan updated the diff for D8900: RunStatusSensorDefinition and @run_status_sensor.

black

Mon, Jul 19, 10:20 PM
yuhan accepted D8947: unpin docstring-parser.

the make build cmd won't hard fail so BK won't be able to capture it. instead, if anything errors in the middle, it'd print out the errors and results empty json output.
if api docs rendering is good, then it's good to go.

Mon, Jul 19, 10:16 PM
yuhan retitled D8900: RunStatusSensorDefinition and @run_status_sensor from PipelineSensorDefinition and @pipeline_sensor to RunStatusSensorDefinition and @run_status_sensor.
Mon, Jul 19, 9:12 PM
yuhan updated the diff for D8900: RunStatusSensorDefinition and @run_status_sensor.
  • run_status_sensor
  • added test to cover the run interleave case for both run-sharded storage and non-run-sharded
Mon, Jul 19, 9:12 PM
yuhan added a comment to D8947: unpin docstring-parser.

can we make sure the make build in docs still works - iirc we had some lib version related issue when building the api docs. i could be wrong but just wanted to make sure unpinning it wont break the docs build.

Mon, Jul 19, 6:08 PM

Fri, Jul 16

yuhan committed R1:0be25391c791: fix algolia config (authored by yuhan).
fix algolia config
Fri, Jul 16, 12:46 AM

Thu, Jul 15

yuhan committed R1:85817ec9e282: remove 0.12.2rc0, 0.12.1rc1 docs version info (authored by yuhan).
remove 0.12.2rc0, 0.12.1rc1 docs version info
Thu, Jul 15, 11:56 PM
yuhan accepted D8907: nav entry for the graph/job/op migration guide.
Thu, Jul 15, 4:54 PM
yuhan published D8907: nav entry for the graph/job/op migration guide for review.
Thu, Jul 15, 4:54 PM

Wed, Jul 14

yuhan added inline comments to D8896: consolidate event type to run status lookup.
Wed, Jul 14, 11:14 PM
yuhan added a comment to D8900: RunStatusSensorDefinition and @run_status_sensor.

For started events, it's expected that I fire the sensor even though by the time I've fired it, the run might have already changed state?

yes bc it's looking at the events not the pipeline statuses, which i think is the desired behavior. alternatively, if we look at the run status, the evaluation may miss firing if the state changes faster than the tick interval.

Wed, Jul 14, 11:13 PM
yuhan requested review of D8900: RunStatusSensorDefinition and @run_status_sensor.
Wed, Jul 14, 10:39 PM
yuhan requested review of D8896: consolidate event type to run status lookup.
Wed, Jul 14, 7:51 PM

Tue, Jul 13

yuhan accepted D8833: make sure pipeline_failure_sensor is under test.

thankyou

Tue, Jul 13, 6:18 PM

Fri, Jul 9

yuhan accepted D8804: 0.12.1 changelog.
Fri, Jul 9, 6:13 PM
yuhan accepted D8801: make get_run_records respect optional mode migration.
Fri, Jul 9, 4:04 PM
yuhan closed D8794: docs search: exclude crawl master/.
Fri, Jul 9, 4:04 PM
yuhan committed R1:cfdf4b20533f: docs search: exclude crawl master/ (authored by yuhan).
docs search: exclude crawl master/
Fri, Jul 9, 4:04 PM
yuhan committed R1:ea0659e62f43: remove 0.12.0rc0 from versioned_content (authored by yuhan).
remove 0.12.0rc0 from versioned_content
Fri, Jul 9, 3:36 AM
yuhan requested review of D8794: docs search: exclude crawl master/.
Fri, Jul 9, 2:36 AM

Thu, Jul 8

yuhan closed D8780: fix master nav in doc build.
Thu, Jul 8, 8:02 PM
yuhan committed R1:5fefb444f179: fix master nav in doc build (authored by yuhan).
fix master nav in doc build
Thu, Jul 8, 8:01 PM
yuhan updated the diff for D8780: fix master nav in doc build.

rebase

Thu, Jul 8, 8:00 PM
yuhan added a reviewer for D8780: fix master nav in doc build: sashank.
Thu, Jul 8, 7:52 PM
yuhan updated the diff for D8780: fix master nav in doc build.

fix sitemap

Thu, Jul 8, 7:49 PM
yuhan published D8780: fix master nav in doc build for review.
Thu, Jul 8, 7:39 PM
yuhan accepted D8778: add better asset sensor docs.
Thu, Jul 8, 7:37 PM
yuhan closed D8769: fix docs.
Thu, Jul 8, 6:04 PM
yuhan committed R1:cd905a925e23: fix docs (authored by alangenfeld).
fix docs
Thu, Jul 8, 6:04 PM
yuhan updated the diff for D8769: fix docs.

disable docs next to unblock bk

Thu, Jul 8, 5:56 PM
yuhan accepted D8769: fix docs.
Thu, Jul 8, 4:25 PM
yuhan published D8769: fix docs for review.
Thu, Jul 8, 4:24 PM
yuhan accepted D8756: fix up internal, event records for get_event_records API.
Thu, Jul 8, 12:11 AM

Wed, Jul 7

yuhan resigned from D8719: [dagit] Better usage of space in RepoNavItem.
Wed, Jul 7, 11:07 PM
yuhan requested changes to D7384: RFC: remove the airline demo.

Q management. lets do it after moving hacker news to public repo

Wed, Jul 7, 9:15 PM
yuhan accepted D8708: docs: remove references to Gitpod.

yes plz

Wed, Jul 7, 9:14 PM
yuhan accepted D8723: add get_mapping_key to SolidExecutionContext.

no context either.. but this looks good to me

Wed, Jul 7, 9:14 PM
yuhan closed D8619: retry from failure fan-in bug.
Wed, Jul 7, 8:01 PM
yuhan committed R1:ecb604a40bef: retry from failure fan-in bug (authored by alangenfeld).
retry from failure fan-in bug
Wed, Jul 7, 8:01 PM
yuhan closed D8722: [docs] composite multiple outputs.
Wed, Jul 7, 7:59 PM
yuhan committed R1:acf7b7c24690: [docs] composite multiple outputs (authored by yuhan).
[docs] composite multiple outputs
Wed, Jul 7, 7:59 PM
yuhan updated the diff for D8619: retry from failure fan-in bug.

rebase

Wed, Jul 7, 6:33 PM
yuhan updated the diff for D8722: [docs] composite multiple outputs.

up

Wed, Jul 7, 6:26 PM
yuhan requested review of D8619: retry from failure fan-in bug.
Wed, Jul 7, 5:00 PM

Tue, Jul 6

yuhan accepted D8691: Add explanation differentiating python types and dagster types.
Tue, Jul 6, 11:12 PM
yuhan accepted D7205: Fix inaccurate repo locations docs.
Tue, Jul 6, 10:49 PM
yuhan resigned from D8660: beef up deploy docker docs.

defer to other reviewers as I'm not qualified enough to review the content here

Tue, Jul 6, 10:48 PM
yuhan added a comment to D8691: Add explanation differentiating python types and dagster types.

This approach makes sense to me. But let's make sure this content doesn't conflict with the "recommended pattern" below https://docs.dagster.io/concepts/types#associating-dagster-types-with-python-types

Tue, Jul 6, 10:47 PM
yuhan published D8722: [docs] composite multiple outputs for review.
Tue, Jul 6, 10:41 PM
yuhan added inline comments to D8695: executor-based IO-management.
Tue, Jul 6, 10:17 PM
yuhan added a comment to D8719: [dagit] Better usage of space in RepoNavItem.

lg2m! defer to others for sign off :)

Tue, Jul 6, 9:15 PM
yuhan closed D8701: RFC: add pipeline_selection to @pipeline_failure_sensor.
Tue, Jul 6, 8:52 PM