up

up

nice!

up

up

up

This stuck out to me a bit when I was reviewing the other diff this morning. lgtm

Looks great! Had two minor comments but otherwise good to land from my perspective

up

added another config typing error change

Overall, I think this is a really strong change. Beyond making the retry story make more sense, sourcing from a URL also makes the configurability make more sense ("hey! now I can just point my pipeline at any cereal csv on the internet!"). It also means that this feels like a full pipeline lifecycle. In the original tutorial, I felt like there was something missing at the front of the pipeline (how do you get this csv onto your computer in the first place), but this feels a lot more complete.

up

up

In D7501#196867, @alangenfeld wrote:

definitely support the overall goal, just need to adjust the approach

this behavior should probably be put under test too

up

Overall, I think this is a positive change. I've personally run into this edge before, and it is frustrating to have to put such a long definition in for an input that has no special properties, simply because a different parameter does.

• restored correct version of file

In D7419#196106, @alangenfeld wrote:

i think this is a clear step forward even if its json sample in yaml land dagit

also - looks like we are not including commas between fields for the "expected" section when there is extra config? not sure if thats an easy diff to throw out while you're in this zone

resolve inlines before landing

up

out of curiosity, where are these >32 bit entries coming from and why does it cause an error in dagit? ideally it would be nice not to silently null out people's metadata. obviously the code here looks correct, just trying to get more context.

love it!

• updated snapshots

rebased

• fixed dead links

• minor edits

• updated graphql test
• fixed prettier
• updated snapshots
• rebased

• added screenshot/text for asset lineage
• removed separate asset catalog page

• restructured asset docs and added content

The list view now shows the asset lineage info for every materialization event. For one of the pipeline runs, I didn't include partitions, so you can see what that looks like as well.

• updated snapshots

• updated snapshots
• added lineage info to list view
• changed heading on partitioned asset pages

• updated graphql test
• fixed prettier
• updated snapshots

In D6910#183831, @sandyryza wrote:

This mostly looks great. A couple thoughts:

• For partitioned assets, users care about the parents of the entire asset, not just the latest partition. Are we able to aggregate that up? It could be in a separate diff if that makes more sense.
• When users click into historical materializations, can we show this information as well?

I added an asset with two parents to the lineage pipeline so that you can see what that looks like.

• cleaned up js and graphene logic

• yarn snapshot

• yarn snapshot

testing

If you apply this diff and and do make dev_webapp in dagster/js_modules/dagit, and then dagit -p 3333 -w workspace.yaml, you can run the asset_lineage pipeline, which shows off the relevant behavior.

fixed more docstrings

I wanted to do my due diligence and see how these integrations would actually look in the real world, so if you're curious:

• refactored hooks and integrated them with the APIs a bit better
• updated docs and added docstrings