Page MenuHomePhabricator

yuhan (Yuhan Luo)
Logistics

Projects

User does not belong to any projects.

User Details

User Since
Feb 28 2020, 11:37 PM (33 w, 5 d)

Recent Activity

Yesterday

yuhan accepted D4859: Fixed a typo.
Wed, Oct 21, 9:57 PM

Tue, Oct 20

yuhan retitled D4579: address-store-1 AddressStore from address-store-1 AddressStore / AssetAddress + AssetStore to address-store-1 AddressStore.
Tue, Oct 20, 11:59 PM
yuhan added a comment to D4820: address-store-0 AssetAddress + AssetStore.
  • I'm skeptical that even supporting path in this is a good idea. I would rather see two asset stores: one that requires paths and one that automatically assigns for you. I think the in between is goofy.

sounds good. will update it.

Tue, Oct 20, 11:17 PM
yuhan updated the diff for D4820: address-store-0 AssetAddress + AssetStore.

arbitrary asset_metadata

Tue, Oct 20, 9:59 PM
yuhan updated the diff for D4820: address-store-0 AssetAddress + AssetStore.

path not required in default asset store

Tue, Oct 20, 9:25 PM
yuhan added inline comments to D4820: address-store-0 AssetAddress + AssetStore.
Tue, Oct 20, 9:25 PM
yuhan added inline comments to D4820: address-store-0 AssetAddress + AssetStore.
Tue, Oct 20, 5:59 PM
yuhan requested review of D4820: address-store-0 AssetAddress + AssetStore.
Tue, Oct 20, 12:50 AM
yuhan planned changes to D4579: address-store-1 AddressStore.
Tue, Oct 20, 12:19 AM
yuhan updated the diff for D4579: address-store-1 AddressStore.

rebase /0

Tue, Oct 20, 12:17 AM

Mon, Oct 19

yuhan updated the summary of D4579: address-store-1 AddressStore.
Mon, Oct 19, 11:48 PM
yuhan requested review of D4810: rfc EphemeralAddressStorage.
Mon, Oct 19, 5:01 PM

Sat, Oct 17

yuhan added a comment to D4719: Revised the Parametrizing Solid with Config in Docs Basics of Solids.

good job! this is looking a lot better

Sat, Oct 17, 6:02 AM

Fri, Oct 16

yuhan added a comment to D4719: Revised the Parametrizing Solid with Config in Docs Basics of Solids.

would be good to show how users can specify the solid config value in [1] and [2] too as you did in the test

Fri, Oct 16, 10:04 PM

Thu, Oct 15

yuhan updated the diff for D4579: address-store-1 AddressStore.

flaky k8s?

Thu, Oct 15, 10:08 PM
yuhan abandoned D4495: [demo-only] asset-intermediate-0 intermediate before vs after.
Thu, Oct 15, 9:53 PM
yuhan added a comment to D4719: Revised the Parametrizing Solid with Config in Docs Basics of Solids.

can you screenshot the final view of the page for reviewing?

Thu, Oct 15, 9:52 PM
yuhan updated the summary of D4579: address-store-1 AddressStore.
Thu, Oct 15, 9:12 PM
yuhan updated the diff for D4579: address-store-1 AddressStore.

make AddressStorage configurale via instance yaml + docstring

Thu, Oct 15, 9:10 PM

Wed, Oct 14

yuhan added a comment to D4579: address-store-1 AddressStore.

@sandyryza

If set_asset is able to determine the address just from the context and the asset_metadata, can get_asset do that as well? I.e. do we need to make a call to a remote service, vs. deriving it from the data we already have available?

did you mean get_asset in AssetStore? what do you mean by a remote service? not sure i fully understand the question πŸ˜…

Wed, Oct 14, 9:00 PM
yuhan added inline comments to D4579: address-store-1 AddressStore.
Wed, Oct 14, 6:46 PM
yuhan updated the diff for D4579: address-store-1 AddressStore.

up

Wed, Oct 14, 6:32 PM
yuhan added inline comments to D4579: address-store-1 AddressStore.
Wed, Oct 14, 5:57 PM
yuhan retitled D4579: address-store-1 AddressStore from address-configurable-intermediate-storage-2 Address + AssetStore + set/get_addressable_asset to address-store-1 AddressStore / AssetAddress + AssetStore.
Wed, Oct 14, 5:57 PM
yuhan updated the diff for D4579: address-store-1 AddressStore.

make AddressStore a new concept on instance -- completely decouple it from intermediate storage

Wed, Oct 14, 5:53 PM

Tue, Oct 13

yuhan planned changes to D4579: address-store-1 AddressStore.

separate from intermediate storage

Tue, Oct 13, 6:50 PM
yuhan added inline comments to D4579: address-store-1 AddressStore.
Tue, Oct 13, 5:29 PM
yuhan updated the summary of D4579: address-store-1 AddressStore.
Tue, Oct 13, 5:25 PM
yuhan planned changes to D4597: address-configurable-intermediate-storage-4 api example RFC.
Tue, Oct 13, 5:19 PM
yuhan planned changes to D4697: address-configurable-intermediate-storage-3 cross-run external_intermediates + pass by reference.
Tue, Oct 13, 5:19 PM
yuhan added inline comments to D4739: asset store discussion pseudo code.
Tue, Oct 13, 5:18 PM
yuhan updated the diff for D4579: address-store-1 AddressStore.

up

Tue, Oct 13, 5:13 PM
yuhan added a comment to D4739: asset store discussion pseudo code.

We get to avoid adding and maintaining an "AddressStore" component that pipes addresses between steps.

in the working prototype of this api D4579, i made address_store a mapping inside IntermediateStorage. i think we can avoid making it an extra layer between intermediates and steps.

Tue, Oct 13, 5:08 PM
yuhan retitled D4579: address-store-1 AddressStore from address-configurable-intermediate-storage-2 `Address` + add address arg to intermediate storage funcs to address-configurable-intermediate-storage-2 Address + AssetStore + set/get_addressable_asset.
Tue, Oct 13, 4:49 PM
yuhan updated the diff for D4579: address-store-1 AddressStore.

AssetAddress, AssetStore, intermediate_storage.address_store

Tue, Oct 13, 4:40 PM

Mon, Oct 12

yuhan added inline comments to D4739: asset store discussion pseudo code.
Mon, Oct 12, 8:42 PM
yuhan updated the summary of D4739: asset store discussion pseudo code.
Mon, Oct 12, 7:29 PM
yuhan updated the diff for D4739: asset store discussion pseudo code.

AssetStore, AddressStore, AssetAddress

Mon, Oct 12, 7:21 PM
yuhan added inline comments to D4738: have IntermediateStorage determine version addresses.
Mon, Oct 12, 5:37 PM

Fri, Oct 9

yuhan added inline comments to D4739: asset store discussion pseudo code.
Fri, Oct 9, 7:28 AM
yuhan added inline comments to D4739: asset store discussion pseudo code.
Fri, Oct 9, 7:25 AM
yuhan updated the summary of D4739: asset store discussion pseudo code.
Fri, Oct 9, 7:25 AM
yuhan published D4739: asset store discussion pseudo code for review.
Fri, Oct 9, 7:12 AM

Thu, Oct 8

yuhan accepted D4735: Fix FileNotFoundError in docs release.
Thu, Oct 8, 11:56 PM
yuhan closed D4726: [docs] mobile view #3054.
Thu, Oct 8, 8:30 PM
yuhan committed R1:58f825ace458: [docs] mobile view #3054 (authored by yuhan).
[docs] mobile view #3054
Thu, Oct 8, 8:30 PM
yuhan published D4726: [docs] mobile view #3054 for review.
Thu, Oct 8, 6:07 PM
yuhan updated the summary of D4697: address-configurable-intermediate-storage-3 cross-run external_intermediates + pass by reference.
Thu, Oct 8, 7:22 AM
yuhan updated the summary of D4597: address-configurable-intermediate-storage-4 api example RFC.
Thu, Oct 8, 7:22 AM
yuhan updated the diff for D4697: address-configurable-intermediate-storage-3 cross-run external_intermediates + pass by reference.

up

Thu, Oct 8, 6:45 AM
yuhan abandoned D4505: asset-storage-1 Asset.

D4579

Thu, Oct 8, 5:42 AM
yuhan abandoned D4506: asset-storage-2 move type loader/materializer into intermediate_storage and add cache.

D4697

Thu, Oct 8, 5:41 AM
yuhan added inline comments to D4562: RFC: lakehouse partitions.
Thu, Oct 8, 5:41 AM
yuhan accepted D4678: factor ObjectStoreOperation creation into IntermediateStorage.
Thu, Oct 8, 5:28 AM

Wed, Oct 7

yuhan updated subscribers of D4677: Replaced outdated images in Parametrizing Solids wiht Config.

are we going to update all the outdated screenshots? can we do a pass at once to update them all together?
or is it something @max 's automated screenshot script can help?

Wed, Oct 7, 9:41 PM
yuhan added inline comments to D4678: factor ObjectStoreOperation creation into IntermediateStorage.
Wed, Oct 7, 4:29 PM
yuhan planned changes to D4495: [demo-only] asset-intermediate-0 intermediate before vs after.
Wed, Oct 7, 3:44 PM
yuhan requested review of D4697: address-configurable-intermediate-storage-3 cross-run external_intermediates + pass by reference.
Wed, Oct 7, 6:41 AM
yuhan updated the diff for D4579: address-store-1 AddressStore.

py2

Wed, Oct 7, 4:44 AM
yuhan retitled D4579: address-store-1 AddressStore from input/output-address-operation-2 add address arg to set/get_intermediate_object and set/get_intermediate to address-configurable-intermediate-storage-2 `Address` + add address arg to intermediate storage funcs.
Wed, Oct 7, 12:37 AM
yuhan retitled D4597: address-configurable-intermediate-storage-4 api example RFC from RFC: address configurable intermediate storage + demo to address-configurable-intermediate-storage-4 api example RFC.
Wed, Oct 7, 12:37 AM
yuhan updated the diff for D4597: address-configurable-intermediate-storage-4 api example RFC.

split diffs

Wed, Oct 7, 12:35 AM

Tue, Oct 6

yuhan updated the diff for D4579: address-store-1 AddressStore.

up

Tue, Oct 6, 11:25 PM
yuhan updated the diff for D4579: address-store-1 AddressStore.

Address

Tue, Oct 6, 11:14 PM
yuhan added inline comments to D4597: address-configurable-intermediate-storage-4 api example RFC.
Tue, Oct 6, 8:04 PM
yuhan updated the diff for D4597: address-configurable-intermediate-storage-4 api example RFC.

update call site in tests (airline_demo and dagster_pandas)

Tue, Oct 6, 7:50 PM
yuhan added a comment to D4678: factor ObjectStoreOperation creation into IntermediateStorage.

this looks good!

Tue, Oct 6, 6:29 PM
yuhan updated the diff for D4597: address-configurable-intermediate-storage-4 api example RFC.

py2 syntax

Tue, Oct 6, 6:20 PM
yuhan retitled D4597: address-configurable-intermediate-storage-4 api example RFC from address configurable intermediate storage + demo to RFC: address configurable intermediate storage + demo.
Tue, Oct 6, 5:44 PM
yuhan accepted D4684: eliminate InMemoryIntermediateStorage.
Tue, Oct 6, 5:13 PM
yuhan updated the summary of D4597: address-configurable-intermediate-storage-4 api example RFC.
Tue, Oct 6, 5:12 PM
yuhan added inline comments to D4597: address-configurable-intermediate-storage-4 api example RFC.
Tue, Oct 6, 5:10 PM
yuhan updated the summary of D4597: address-configurable-intermediate-storage-4 api example RFC.
Tue, Oct 6, 5:06 PM
yuhan updated the diff for D4597: address-configurable-intermediate-storage-4 api example RFC.
  • cross-run intermediates based on events -> reexecution works
  • event - SET_EXTERNAL_OBJECT, GET_EXTERNAL_OBJECT
  • example - pass address from Output inside a solid
Tue, Oct 6, 4:46 PM
yuhan updated the diff for D4597: address-configurable-intermediate-storage-4 api example RFC.

limitation - dagster type loader and materializer has to be paired if they are used for intermediate operations

Tue, Oct 6, 6:03 AM
yuhan added inline comments to D4597: address-configurable-intermediate-storage-4 api example RFC.
Tue, Oct 6, 5:17 AM
yuhan updated the diff for D4597: address-configurable-intermediate-storage-4 api example RFC.

simplify examples - use solid_config.outputs to handle intermediates

Tue, Oct 6, 5:12 AM
yuhan updated the diff for D4597: address-configurable-intermediate-storage-4 api example RFC.

simplify examples - use solid_config.outputs to handle intermediates

Tue, Oct 6, 5:09 AM
yuhan accepted D4676: fix some outstanding typo and wording.
Tue, Oct 6, 3:41 AM

Fri, Oct 2

yuhan added inline comments to D4597: address-configurable-intermediate-storage-4 api example RFC.
Fri, Oct 2, 9:43 PM
yuhan requested review of D4597: address-configurable-intermediate-storage-4 api example RFC.
Fri, Oct 2, 8:27 PM
yuhan accepted D4570: partition asset materializations in longitudinal pipeline.
Fri, Oct 2, 6:19 PM
yuhan accepted D4598: [lakehouse] accept an optional "context" arg for computations.

I kinda feel it's less robust to surface the variable by the literal arg name "context" and it could be consistent to all other entities (e.g. solid, resource, hook, etc) if the context is always the first positional arg.
my general sense was it's better to be restrict first and then lose the constraint than the opposite. I don't feel strongly about it in this case tho -- i trust your call

Fri, Oct 2, 6:17 PM
yuhan accepted D4638: include SnowflakeConnection in API docs.
Fri, Oct 2, 6:08 PM

Thu, Oct 1

yuhan planned changes to D4579: address-store-1 AddressStore.
Thu, Oct 1, 4:55 PM
yuhan accepted D4605: toy pyspark pipeline.

great start!

Thu, Oct 1, 2:14 AM

Wed, Sep 30

yuhan added a comment to D4598: [lakehouse] accept an optional "context" arg for computations.

should we be more restrict to limit context to be a required arg and at the first position like all other decorators do?

Wed, Sep 30, 7:28 PM
yuhan added inline comments to D4526: AssetMaterialization partitions.
Wed, Sep 30, 7:26 PM

Tue, Sep 29

yuhan accepted D4592: Adds scrolling and line wrapping to Examples index page..
Tue, Sep 29, 8:32 PM
yuhan closed D4578: input/output-address-operation-1 intermediate storage set/get from address.
Tue, Sep 29, 7:14 PM
yuhan committed R1:d8e42ee57dac: input/output-address-operation-1 intermediate storage set/get from address (authored by yuhan).
input/output-address-operation-1 intermediate storage set/get from address
Tue, Sep 29, 7:14 PM
yuhan updated the diff for D4579: address-store-1 AddressStore.

experimental_arg_warning

Tue, Sep 29, 7:14 PM
yuhan updated the diff for D4578: input/output-address-operation-1 intermediate storage set/get from address.

experimental + DagsterInvalidAddressForAsset -> DagsterAddressIOError

Tue, Sep 29, 6:42 PM
yuhan added inline comments to D4578: input/output-address-operation-1 intermediate storage set/get from address.
Tue, Sep 29, 5:10 PM
yuhan accepted D3882: Make re-execute a two-part button so it’s one-click by default #2687.

this looks good to me.

Tue, Sep 29, 4:03 PM
yuhan added inline comments to D4577: added chars and test for asset key validation. fixed errors thrown in `validate_asset_key_string` and `validate_structured_asset_key`, as they were not thrown correctly and would error out if reached..
Tue, Sep 29, 3:51 PM
yuhan added inline comments to D4578: input/output-address-operation-1 intermediate storage set/get from address.
Tue, Sep 29, 2:56 PM
yuhan updated the summary of D4579: address-store-1 AddressStore.
Tue, Sep 29, 2:55 PM
yuhan added inline comments to D4577: added chars and test for asset key validation. fixed errors thrown in `validate_asset_key_string` and `validate_structured_asset_key`, as they were not thrown correctly and would error out if reached..
Tue, Sep 29, 2:52 PM
yuhan requested review of D4579: address-store-1 AddressStore.
Tue, Sep 29, 2:45 PM