Page MenuHomeElementl

yuhan (Yuhan Luo)
User

Projects

User does not belong to any projects.

User Details

User Since
Feb 28 2020, 11:37 PM (85 w, 6 d)

Recent Activity

Fri, Oct 8

yuhan committed R1:7a2373febf5d: versioned docs for 0.12.14 (authored by yuhan).
versioned docs for 0.12.14
Fri, Oct 8, 1:06 AM

Sep 3 2021

yuhan committed R1:82c842e77a85: Automation: versioned docs for 0.12.9 (authored by elementl-devtools).
Automation: versioned docs for 0.12.9
Sep 3 2021, 8:11 AM

Aug 16 2021

yuhan accepted D9227: fix dagster-mlflow hook.
Aug 16 2021, 11:46 PM

Aug 12 2021

yuhan abandoned D6145: Use IO managers for Dagstermill serialization.

Finishing this diff in the following PRs:

Aug 12 2021, 6:46 AM

Aug 2 2021

yuhan accepted D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

my hunch is later in the stack (after D9085), this shouldn't be too large of a change, e.g. could potentially toggle is on io manager config?

Aug 2 2021, 5:18 PM

Jul 30 2021

yuhan added a comment to D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

in terms of user experience, i think it's fine to implicitly add a tag to a run - some other features work like that, e.g. we implicitly add parent_run_id and root_run_id as tags to new runs, and solid/step selection too. besides, it's also good that users can view those info in the Runs page such as:

image.png (802×2 px, 185 KB)

Jul 30 2021, 7:13 PM
yuhan added a comment to D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

Which brings us back to Alex's initial q: what's the right way to toggle memoization on and off?

Jul 30 2021, 6:23 PM
yuhan commandeered D6145: Use IO managers for Dagstermill serialization.
Jul 30 2021, 5:37 PM
yuhan planned changes to D8981: add `from_failure` bool arg to rexecute pipeline apis.

trying enum

Jul 30 2021, 5:12 PM
yuhan published D4698: Prototype: Add test utility for dagstermill solids for review.
Jul 30 2021, 12:17 AM

Jul 29 2021

yuhan added a comment to D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

in terms of modeling similar "load-from-other-run-in-the-run-group look-ups" in the same way, i could see a path forward that we consolidate logics into something like get_retry_steps_from_execution_plan - at a high level, all the paths are basically fn(historical execution plann, logs) -> execution_plan.

Jul 29 2021, 7:46 PM
yuhan accepted D9059: [memoization 2/n] provide default base dir for versioned fs io manager.
Jul 29 2021, 7:26 PM
yuhan accepted D9144: [dagit] Repair repo specificity on Run details and re-execution.
Jul 29 2021, 6:14 PM

Jul 28 2021

yuhan closed D9110: dagstermill - yield_result output path account for aliased solids and composite solids.
Jul 28 2021, 11:23 PM
yuhan committed R1:f1c0a4f53e57: dagstermill - yield_result output path account for aliased solids and composite… (authored by yuhan).
dagstermill - yield_result output path account for aliased solids and composite…
Jul 28 2021, 11:23 PM
yuhan requested review of D9110: dagstermill - yield_result output path account for aliased solids and composite solids.
Jul 28 2021, 4:37 AM

Jul 27 2021

yuhan closed D9071: mypy run storage.
Jul 27 2021, 11:08 PM
yuhan committed R1:bc9d4942212f: mypy run storage (authored by yuhan).
mypy run storage
Jul 27 2021, 11:08 PM
yuhan accepted D9103: Add missing imports to hacker_news example repo.py.
Jul 27 2021, 11:02 PM
yuhan updated the diff for D9071: mypy run storage.

gql

Jul 27 2021, 10:16 PM
yuhan updated the diff for D9071: mypy run storage.

loosen handle_run_event error
add error handling to in mem storage

Jul 27 2021, 8:43 PM
yuhan added inline comments to D9059: [memoization 2/n] provide default base dir for versioned fs io manager.
Jul 27 2021, 6:56 PM
yuhan closed D9017: [docs] sensor concept include run status sensor.
Jul 27 2021, 6:47 PM
yuhan committed R1:67442bdea2da: [docs] sensor concept include run status sensor (authored by yuhan).
[docs] sensor concept include run status sensor
Jul 27 2021, 6:47 PM
yuhan added a comment to D9017: [docs] sensor concept include run status sensor.

yup pipeline failure sensor is already nested under the run status sensor and solid hooks are still linked - i didn't update the failure sensor section in this diff.

Jul 27 2021, 6:46 PM
yuhan updated the diff for D9071: mypy run storage.

DagsterRunNotFoundError

Jul 27 2021, 6:41 PM
yuhan closed D9053: dagstermill - fix: yield_result temp path includes solid name.
Jul 27 2021, 5:35 PM
yuhan committed R1:ad5abf66f0b2: dagstermill - fix: yield_result temp path includes solid name (authored by yuhan).
dagstermill - fix: yield_result temp path includes solid name
Jul 27 2021, 5:35 PM
yuhan requested review of D9071: mypy run storage.
Jul 27 2021, 2:09 AM

Jul 26 2021

yuhan requested review of D9053: dagstermill - fix: yield_result temp path includes solid name.
Jul 26 2021, 6:16 PM

Jul 23 2021

yuhan accepted D9000: add title and description to graph migration page.
Jul 23 2021, 11:25 PM
yuhan accepted D9013: Add dbt pipeline to example repo, synchronize with internal.

this is great!

Jul 23 2021, 11:24 PM
yuhan accepted D9047: Move invalid step logic down into gRPC, to reduce gRPC calls.
Jul 23 2021, 11:00 PM
yuhan accepted D9035: tailwind config for highlighting back links in source snippets.
Jul 23 2021, 1:18 AM

Jul 22 2021

yuhan requested review of D9017: [docs] sensor concept include run status sensor.
Jul 22 2021, 7:53 PM
yuhan added inline comments to D9021: changes for 0.12.3.
Jul 22 2021, 7:11 PM
yuhan closed D9015: run_status_sensor api doc.
Jul 22 2021, 6:26 PM
yuhan committed R1:1b305b5db8d7: run_status_sensor api doc (authored by yuhan).
run_status_sensor api doc
Jul 22 2021, 6:26 PM
yuhan closed D8900: RunStatusSensorDefinition and @run_status_sensor.
Jul 22 2021, 5:43 PM
yuhan committed R1:306593aa82c9: RunStatusSensorDefinition and @run_status_sensor (authored by yuhan).
RunStatusSensorDefinition and @run_status_sensor
Jul 22 2021, 5:43 PM
yuhan closed D8896: consolidate event type to run status lookup.
Jul 22 2021, 5:29 PM
yuhan committed R1:19f525d6f7be: consolidate event type to run status lookup (authored by yuhan).
consolidate event type to run status lookup
Jul 22 2021, 5:29 PM
yuhan added inline comments to D8900: RunStatusSensorDefinition and @run_status_sensor.
Jul 22 2021, 12:35 AM
yuhan updated the diff for D8900: RunStatusSensorDefinition and @run_status_sensor.

PipelineSensorCursor -> RunStatusSensorCursor + backcompat fallback

Jul 22 2021, 12:35 AM

Jul 21 2021

yuhan added inline comments to D8900: RunStatusSensorDefinition and @run_status_sensor.
Jul 21 2021, 11:33 PM
yuhan added a comment to D8981: add `from_failure` bool arg to rexecute pipeline apis.

one alternative would be to extend step_selection to also take an enum entry or marker type for describing a step selection to be calculated by dagster.

we could

  • extend step_selection to take StepSelection.FROM_FAILURE), step_selection=StepSelection.from_selection(["some_solid*"]), and ["some_solid*"] (backcompat)
  • or a more generic arg reexecute_option and deprecate step_selection, it'd take ReexecuteOption.FROM_FAILURE, ReexecuteOption.from_selection(["some_solid*"]))
Jul 21 2021, 9:23 PM
yuhan requested review of D8981: add `from_failure` bool arg to rexecute pipeline apis.
Jul 21 2021, 6:54 PM

Jul 20 2021

yuhan accepted D8980: Fix sensor testing docs.
Jul 20 2021, 11:38 PM
yuhan accepted D8976: [new-project] sensor targets non-existent mode.

not directly related to this diff: how do people feel about having a separate new-project cli for the crag stuff?

Jul 20 2021, 11:23 PM
yuhan updated the diff for D8900: RunStatusSensorDefinition and @run_status_sensor.

PipelineSensor -> RunStatusSensor

Jul 20 2021, 11:18 PM
yuhan added inline comments to D8900: RunStatusSensorDefinition and @run_status_sensor.
Jul 20 2021, 11:12 PM
yuhan accepted D8974: remove bad use of 'in'.
Jul 20 2021, 9:03 PM

Jul 19 2021

yuhan updated the diff for D8900: RunStatusSensorDefinition and @run_status_sensor.

black

Jul 19 2021, 10:20 PM
yuhan accepted D8947: unpin docstring-parser.

if it's version conflict, the make build cmd won't hard fail so BK won't be able to capture it. instead, if anything errors in the middle, it'd print out the errors and results empty json output for the ones that has conflict.
if api docs rendering is good, then it's good to go.

Jul 19 2021, 10:16 PM
yuhan retitled D8900: RunStatusSensorDefinition and @run_status_sensor from PipelineSensorDefinition and @pipeline_sensor to RunStatusSensorDefinition and @run_status_sensor.
Jul 19 2021, 9:12 PM
yuhan updated the diff for D8900: RunStatusSensorDefinition and @run_status_sensor.
  • run_status_sensor
  • added test to cover the run interleave case for both run-sharded storage and non-run-sharded
Jul 19 2021, 9:12 PM
yuhan added a comment to D8947: unpin docstring-parser.

can we make sure the make build in docs still works - iirc we had some lib version related issue when building the api docs. i could be wrong but just wanted to make sure unpinning it wont break the docs build.

Jul 19 2021, 6:08 PM

Jul 16 2021

yuhan committed R1:0be25391c791: fix algolia config (authored by yuhan).
fix algolia config
Jul 16 2021, 12:46 AM

Jul 15 2021

yuhan committed R1:85817ec9e282: remove 0.12.2rc0, 0.12.1rc1 docs version info (authored by yuhan).
remove 0.12.2rc0, 0.12.1rc1 docs version info
Jul 15 2021, 11:56 PM
yuhan accepted D8907: nav entry for the graph/job/op migration guide.
Jul 15 2021, 4:54 PM
yuhan published D8907: nav entry for the graph/job/op migration guide for review.
Jul 15 2021, 4:54 PM

Jul 14 2021

yuhan added inline comments to D8896: consolidate event type to run status lookup.
Jul 14 2021, 11:14 PM
yuhan added a comment to D8900: RunStatusSensorDefinition and @run_status_sensor.

For started events, it's expected that I fire the sensor even though by the time I've fired it, the run might have already changed state?

yes bc it's looking at the events not the pipeline statuses, which i think is the desired behavior. alternatively, if we look at the run status, the evaluation may miss firing if the state changes faster than the tick interval.

Jul 14 2021, 11:13 PM
yuhan requested review of D8900: RunStatusSensorDefinition and @run_status_sensor.
Jul 14 2021, 10:39 PM
yuhan requested review of D8896: consolidate event type to run status lookup.
Jul 14 2021, 7:51 PM

Jul 13 2021

yuhan accepted D8833: make sure pipeline_failure_sensor is under test.

thankyou

Jul 13 2021, 6:18 PM

Jul 9 2021

yuhan accepted D8804: 0.12.1 changelog.
Jul 9 2021, 6:13 PM
yuhan accepted D8801: make get_run_records respect optional mode migration.
Jul 9 2021, 4:04 PM
yuhan closed D8794: docs search: exclude crawl master/.
Jul 9 2021, 4:04 PM
yuhan committed R1:cfdf4b20533f: docs search: exclude crawl master/ (authored by yuhan).
docs search: exclude crawl master/
Jul 9 2021, 4:04 PM
yuhan committed R1:ea0659e62f43: remove 0.12.0rc0 from versioned_content (authored by yuhan).
remove 0.12.0rc0 from versioned_content
Jul 9 2021, 3:36 AM
yuhan requested review of D8794: docs search: exclude crawl master/.
Jul 9 2021, 2:36 AM

Jul 8 2021

yuhan closed D8780: fix master nav in doc build.
Jul 8 2021, 8:02 PM
yuhan committed R1:5fefb444f179: fix master nav in doc build (authored by yuhan).
fix master nav in doc build
Jul 8 2021, 8:01 PM
yuhan updated the diff for D8780: fix master nav in doc build.

rebase

Jul 8 2021, 8:00 PM
yuhan added a reviewer for D8780: fix master nav in doc build: sashank.
Jul 8 2021, 7:52 PM
yuhan updated the diff for D8780: fix master nav in doc build.

fix sitemap

Jul 8 2021, 7:49 PM
yuhan published D8780: fix master nav in doc build for review.
Jul 8 2021, 7:39 PM
yuhan accepted D8778: add better asset sensor docs.
Jul 8 2021, 7:37 PM
yuhan closed D8769: fix docs.
Jul 8 2021, 6:04 PM
yuhan committed R1:cd905a925e23: fix docs (authored by alangenfeld).
fix docs
Jul 8 2021, 6:04 PM
yuhan updated the diff for D8769: fix docs.

disable docs next to unblock bk

Jul 8 2021, 5:56 PM
yuhan accepted D8769: fix docs.
Jul 8 2021, 4:25 PM
yuhan published D8769: fix docs for review.
Jul 8 2021, 4:24 PM
yuhan accepted D8756: fix up internal, event records for get_event_records API.
Jul 8 2021, 12:11 AM

Jul 7 2021

yuhan resigned from D8719: [dagit] Better usage of space in RepoNavItem.
Jul 7 2021, 11:07 PM
yuhan requested changes to D7384: RFC: remove the airline demo.

Q management. lets do it after moving hacker news to public repo

Jul 7 2021, 9:15 PM
yuhan accepted D8708: docs: remove references to Gitpod.

yes plz

Jul 7 2021, 9:14 PM
yuhan accepted D8723: add get_mapping_key to SolidExecutionContext.

no context either.. but this looks good to me

Jul 7 2021, 9:14 PM
yuhan closed D8619: retry from failure fan-in bug.
Jul 7 2021, 8:01 PM
yuhan committed R1:ecb604a40bef: retry from failure fan-in bug (authored by alangenfeld).
retry from failure fan-in bug
Jul 7 2021, 8:01 PM
yuhan closed D8722: [docs] composite multiple outputs.
Jul 7 2021, 7:59 PM
yuhan committed R1:acf7b7c24690: [docs] composite multiple outputs (authored by yuhan).
[docs] composite multiple outputs
Jul 7 2021, 7:59 PM
yuhan updated the diff for D8619: retry from failure fan-in bug.

rebase

Jul 7 2021, 6:33 PM
yuhan updated the diff for D8722: [docs] composite multiple outputs.

up

Jul 7 2021, 6:26 PM
yuhan requested review of D8619: retry from failure fan-in bug.
Jul 7 2021, 5:00 PM

Jul 6 2021

yuhan accepted D8691: Add explanation differentiating python types and dagster types.
Jul 6 2021, 11:12 PM
yuhan accepted D7205: Fix inaccurate repo locations docs.
Jul 6 2021, 10:49 PM
yuhan resigned from D8660: beef up deploy docker docs.

defer to other reviewers as I'm not qualified enough to review the content here

Jul 6 2021, 10:48 PM
yuhan added a comment to D8691: Add explanation differentiating python types and dagster types.

This approach makes sense to me. But let's make sure this content doesn't conflict with the "recommended pattern" below https://docs.dagster.io/concepts/types#associating-dagster-types-with-python-types

Jul 6 2021, 10:47 PM