ready for a once over! Good call on the naming change and the container switch, this actually makes my general life a lot easier as well since I switched my dagster-home to point to the test-postgres-db.

"risky" might be the wrong word but more like "too easy to break" meaning that anytime you change that config key it will certainly not behave the way you want to. Any change with the config would have to be coupled with a change to code. Given that it doesn't seem like something that should be in config

Exposing a configurable priority key is actually pretty risky and will result in a lot of replicated config.

• renamed dagster_components to common and switched to using the test-postgres-db container

So the crux of it is:

• we do dagster/priority and it works for the core executors but (probably?) none of the others which leads to an unfortunate expectation mis-match in a not clearly communicated way (keys just ignored)
• we do separate keys for each executor to align expectations ie dagster/in-process/priority, dagster-in-process/priority, whatever but thats aesthetically unfortunate and leads to having to set all keys to support different execution modes

priority?

what should the predefined keys be for the in process and multi process engines? I wasn't happy with what I came up with which motivated me to punt it out to config

So what I was expecting to see here is not a configurable priority key in terms of *configuring* executor on a per-run basis, but instead just having it be part of the executor definition, plus a comparator functions between priorities. Exposing a configurable priority key is actually pretty risky and will result in a lot of replicated config.

Rebase

Nits

blk

move sorting out to individual engines. use config to drive what key to use for prioirty on default engines

py27

rebase

as in, not having the migration seems fine, when you can accomplish the same thing by running wipe and up

I think it should be fine, but we should add something to the changelog, saying "Need to run dagster schedule ..."

Fixup py27

Up

Up

up

partition stuff looks good.

Fix merge conflict

This shouldn't break existing running schedules because the scheduler manages those all by name, correct?

Add comment about why we need to do scheduler_handle.up

This shouldn't break existing running schedules because the scheduler manages those all by name, correct?

Rebase again

Rebase on master

changes here all seem reasonable to me

I think this is better

rebase

dont have strong feelings - but the complexity cost of warn here feels pretty minimal so i maybe lean that way

cool. it's not too bad. where do you land on breaking vs warn?

can you explain automated comment. i think i am missing something

A couple pieces of feedback you can address here or in a follow-on:

Abandoning for now.

How difficult would it be to make this not required a full reload? E.g. have the flatten composites button/check stay in view while only dag view reloads. Extra credit for some sort of animation or fade.

• tweaked test because c/i has no api key installed

rebase on master

handle small button sizes correctly

rebase

Realized that if we wanted to switch to the partition API, we need to design my pipelines to require transactional consistency. cloud storage is not that, so we need to switch to using a database. I should have done this a long time ago, but am going to do a bunch of incremental refactors to get there instead of blowing up this diff.

prevent local storage clobber

rebase

Repushing to bring in buildkite nbconvert issues.

Fix AppCache dataIdFromObject for RunningSchedule

LGTM

upmessage