Disable dask tests again

• got rid of silly name keyword param and made it required

I'm quite happy with this direction. curious what you think @alangenfeld

still need to make name required. name=None will throw. a test should confirm this

• made docs fixes and added comment

couple things.... still trying to figure out the exception handling. I think we weren't tearing down resources properly before, so figuring out what tests we need to write, change, etc.

• pipeline initialization events, but with generators INSTEAD of context managers

just getting this off my laptop before commute, will update further tomorrow

WIP on discussed changes

req'ing changes based on name being optional

would recommend using arc patch in that situation

Holy shit. In all the excitement, I realized I never checked out a branch....uhhhh gonna abandon this and get a new revision out with the fixes. I'll link this revision to that one but will need another LGTM

I'd prefer not to take the dependency but will defer to others

• rebasing to pick up buildkite changes.

cool

On it.

looks good. plz heed final comments!

yeah we could do something like that, although for secrets we'll use an entirely different k8s object, Secret https://kubernetes.io/docs/concepts/configuration/secret/ instead of a ConfigMap—so instead of the previous approach, we'd do like DAGSTER_PG_PASSWORD above—see [1]

is this env: approach going to support loading secrets in as environment variables? The the env_from approach supports that right? Should we have a base then dagit and job_runner specific env_from that defaults to something populated by the env: approach? That way the overriding behavior is handled by env_from load order on the pod?

have an explicit base and then the two override sections

yeah what I'm suggesting is just to have the base map of env vars and drop support for overrides; the dagit/job-specific overrides _can_ work, but e.g. if I have:

We could have one global Dagster ConfigMap to spec extra env vars for everywhere (dagit, celery workers, job containers)—or, as in this diff, separate env ConfigMaps for dagit vs. celery/job nodes. I lean towards the former to keep things simple, but lmk

• renamed to more explicit val

@alangenfeld RFC here

up

Looks reasonable to me; only question I have here is whether we should just use tenacity (https://tenacity.readthedocs.io/en/latest/) vs. rolling our own—thoughts?

Regardless of whether pandas' API changes in the future, we need to be compatible with all versions of pandas which makes it difficult to have a strictly typed schema right? I mean it's a tradeoff, but I don't know if we want to tie ourselves to maintaining it.

• addressed feedback

Pandas is at 1.0 now -- https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.0.0.html

Up

Six

Rebase

Resolves https://github.com/dagster-io/dagster/issues/2174

fix scoped_pipeline_context reference

add try/finally, remove double defaults

I initially had it so that the resource/builder/context were yielded along with the event stream, but switched it (per @alangenfeld's suggestion) because it makes it way more explicit in the caller what is going on. I do like this better and we're not mixing types so much. It was getting confusing where we're stacking things at different levels and the type I'm trying to extract is different at each level.

cool stuff. like how hard you are pushing on modes/testing

This seems like a good direction. Great job wading through this. Thorny part of the codebase for sure.

rebase

Rebase

Move to branch

rebase

Nice improvement 👍

rebase

• brought in LocalClient resource

rebase

Done because I realized I need to make a core change.

• rebased with master to pick up buildkite changes