Page MenuHomePhabricator

[RunQueue] LocalComputeQueue
AbandonedPublic

Authored by alangenfeld on Oct 17 2019, 7:22 PM.

Details

Reviewers
None
Group Reviewers
Restricted Project
Summary

Create a RunQueue that manages the local compute resource - limiting the number of active processes.

Test Plan

test case

Diff Detail

Repository
R1 dagster
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

alangenfeld created this revision.Oct 17 2019, 7:22 PM
alangenfeld planned changes to this revision.Oct 17 2019, 8:02 PM

still a bit rough - putting up early so people can see progress and make suggestions

alangenfeld updated this revision to Diff 5887.Oct 17 2019, 9:31 PM

rebase, include delegating queue

prha added a subscriber: prha.Oct 22 2019, 5:04 PM

Couple questions, but overall I think this looks good! Will let max and schrockn take a look also.

python_modules/dagster-graphql/dagster_graphql_tests/graphql/test_run_queue.py
104

maybe a helper function that takes in the different queue configs and runs the test?

Took a while to confirm the difference between test_local_compute_queue and test_delegating_queue

python_modules/dagster/dagster/core/run_queue/delegating.py
12

do you think DelegatingRunQueue will be a common case? Is this mostly to set up different priority queues?

python_modules/dagster/dagster/core/run_queue/local_compute.py
95

should we do a check here for non-null PipelineRun to guarantee that we don't accidentally stop the run queue processing?

alangenfeld added inline comments.Oct 22 2019, 5:53 PM
python_modules/dagster-graphql/dagster_graphql_tests/graphql/test_run_queue.py
104

the test bodies are a bit different [1]

140–142

[1]

python_modules/dagster/dagster/core/run_queue/delegating.py
12

not sure how common - but ya so you could set up different queues for different resource requirements ie dagster/queue/needs_gpu

alangenfeld planned changes to this revision.Oct 28 2019, 4:25 PM
alangenfeld abandoned this revision.Tue, Dec 3, 4:38 PM