accepting because of fun tests in other diff

1. We should definitely configure phab to hide those huge data files
2. This is really cool! Can you post a screenshot of the visual diff? Really curious what that looks like.

Yeah agree that it's not a slam dunk

Actually I'm quite wrong given that the EventLogStorage class in question has context about the underlying store.

agree on deciding on some sort of namespacing scheme here. i bias towards {'dask': {'resource_requirements': {...}}} option because it is inline with our current config system. but it does say something the software people like (e.g. vscode and k8s) using the in-key namespacing scheme

Sauce

This is a solid step forward. I'd love for you to comment on this diff re: what next steps are in terms of reining it in. I assume we want to get rid of the PipelineRun hierarchy?

This is looking good. Per discussion in follow ons let's extract dagster-cron (or whatever we call it) into its own module so that the core library is not tied to the cron scheduler.

ya this seems nicer

Exciting stuff. I barely know anything about systems-level linux issues, so I can't really give substantive feedback on high level stuff, but this all seems reasonable.

This is really exciting to see. Just have the request about restart latency.

I'd also love some screenshots from pipelines generated with https://dagster.phacility.com/D846

per IRL discussion let's do some better names and comments because this is complicated, but looks good

up

Result of construct_level_pipeline('fan_in_fan_out_pipeline', levels=10, fanout=50)

Let's get this merged. Please heed final feedback

seems good now. will let @alangenfeld approve

decorator is nice

conn instead of provider

lovely

So will limit what versions of airflow we can integrate with? I’m a little hazy on how this all coheres to a flexible sane strategy around deps

This looks great to me. I'll let @max approve to resolve your question re: the use/non-use of raise_from.

up

self review

py2 floats

py2 floats

up

@alangenfeld curious about what you think should be our standard for passing around connection strings given the py2-only bs you ran into.

This is quite lovely. @max should take a look prior to merging but this looks great.

cool

can you screenshot output of hello world tutorial?

up

rebase

comment

this magically fixed itself. still not sure what happened.

I'm getting this error in a dev build with this diff patched.

what does dagster cli look like on hello world?

black

py2 and test

Maybe we should have some be debug and some be info

Hmmm. I think maybe we should upgrade the emitted system events to INFO instead?

would be cool to add a test that catches this

So think we might to have this be a row-per required resource. That way we can add things like descriptions or metadata to the resource requirement.

nice very surgical

Here is a video of this in action.

cool

great let's move forward. seems like next critical thing here will be to set up an integration test

up

thx

this looks reasonable overall. if python modules were easier to manage for us i might push for a separate module for this but meh for now.

just popping off of my queue

seems like a good start. will let @alangenfeld do final approval as you two have been discussing

👍🏻

up

up