Page MenuHomePhabricator
Differential D4674

Add liveness probe to extra celery workers
ClosedPublic
Actions

Authored by rexledesma on Oct 6 2020, 12:55 AM.

Details

Reviewers
johann
nate
catherinewu
Commits
R1:b522ecc3a69f: Add liveness probe to extra celery workers
Summary

As the title.

Test Plan

bk, add fields to helm config

Diff Detail

Repository
R1 dagster
Branch
arcpatch-D4674
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

rexledesma created this revision.Oct 6 2020, 12:55 AM
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 6 2020, 1:39 AM
Harbormaster failed remote builds in B19167: Diff 23294!
rexledesma updated this revision to Diff 23333.Oct 6 2020, 6:53 PM

up

rexledesma updated this revision to Diff 23338.Oct 6 2020, 7:19 PM

up

rexledesma updated this revision to Diff 23341.Oct 6 2020, 7:27 PM

skip everything except integration tests

Harbormaster failed remote builds in B19205: Diff 23338!Oct 6 2020, 7:38 PM
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 6 2020, 7:47 PM
Harbormaster failed remote builds in B19208: Diff 23341!
Harbormaster failed remote builds in B19201: Diff 23333!Oct 6 2020, 7:59 PM
rexledesma updated this revision to Diff 23344.Oct 6 2020, 8:00 PM

up

rexledesma updated this revision to Diff 23347.Oct 6 2020, 8:25 PM

up

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 6 2020, 8:44 PM
Harbormaster failed remote builds in B19213: Diff 23347!
rexledesma updated this revision to Diff 23353.Oct 6 2020, 9:00 PM

up

Harbormaster failed remote builds in B19210: Diff 23344!Oct 6 2020, 9:09 PM
Harbormaster completed remote builds in B19217: Diff 23353.Oct 6 2020, 9:16 PM
rexledesma requested review of this revision.Oct 6 2020, 9:16 PM
rexledesma updated this revision to Diff 23357.Oct 6 2020, 9:26 PM

up

Harbormaster failed remote builds in B19221: Diff 23357!Oct 6 2020, 9:55 PM
rexledesma updated this revision to Diff 23361.Oct 6 2020, 10:25 PM

up

Harbormaster completed remote builds in B19224: Diff 23361.Oct 6 2020, 10:41 PM
rexledesma updated this revision to Diff 23363.Oct 6 2020, 10:54 PM

up

Harbormaster failed remote builds in B19226: Diff 23363!Oct 6 2020, 11:13 PM
rexledesma updated this revision to Diff 23376.Oct 6 2020, 11:54 PM

rebase

Harbormaster failed remote builds in B19239: Diff 23376!Oct 7 2020, 12:25 AM
rexledesma edited the test plan for this revision. (Show Details)Oct 7 2020, 12:38 AM
rexledesma added reviewers: johann, nate, catherinewu.
Harbormaster completed remote builds in B19239: Diff 23376.Oct 7 2020, 1:13 AM
johann added a subscriber: alangenfeld.Oct 12 2020, 9:01 PM
johann added inline comments.
integration_tests/python_modules/dagster-k8s-test-infra/dagster_k8s_test_infra/helm.py
336

@alangenfeld - you mentioned you only wanted to have the extra worker for a single test. The cluster is session scoped though, so I'm not sure how we'd do that

catherinewu added a comment.Oct 12 2020, 9:59 PM

Since the extraWorkerQueues part of this test has caused flakiness in the past, can we make sure to run this through bk a few times (5?) times before merging? last time i tried to add extraWorkerQueues back, it passed the first three runs and then failed twice (https://dagster.phacility.com/D4535)

alangenfeld added inline comments.Oct 13 2020, 2:47 PM
integration_tests/python_modules/dagster-k8s-test-infra/dagster_k8s_test_infra/helm.py
336

The cluster is session scoped

Ah i see I didn't realize this - not sure whats best. You could do a helm upgrade with new chunk of values but thats pretty complicated

rexledesma updated this revision to Diff 23839.Oct 16 2020, 1:48 AM

bk#2

rexledesma updated this revision to Diff 23840.Oct 16 2020, 1:55 AM

bk#3

rexledesma updated this revision to Diff 23841.Oct 16 2020, 1:57 AM

bk#4

rexledesma updated this revision to Diff 23842.Oct 16 2020, 1:57 AM

bk#5

Harbormaster completed remote builds in B19629: Diff 23839.Oct 16 2020, 2:08 AM
Harbormaster completed remote builds in B19630: Diff 23840.Oct 16 2020, 2:15 AM
Harbormaster completed remote builds in B19631: Diff 23841.
Harbormaster completed remote builds in B19632: Diff 23842.Oct 16 2020, 2:19 AM
catherinewu accepted this revision.Oct 16 2020, 8:06 PM

awesome, looks good!

Macro pooh_eating_honey:

after we merge celery and celery-extras yaml, we probably want to make it possible to configure different liveness probes per queue

This revision is now accepted and ready to land.Oct 16 2020, 8:06 PM
Closed by commit R1:b522ecc3a69f: Add liveness probe to extra celery workers (authored by rexledesma). · Explain WhyOct 19 2020, 11:48 PM
This revision was automatically updated to reflect the committed changes.
rexledesma added a commit: R1:b522ecc3a69f: Add liveness probe to extra celery workers.

Revision Contents
Changeset List

PathSize
helm/
dagster/
templates/
33 lines
2 lines
integration_tests/
python_modules/
dagster-k8s-test-infra/
dagster_k8s_test_infra/
16 lines

Diff 23840

View Options

helm/dagster/templates/deployment-celery-extras.yaml

Loading...
View Options

helm/dagster/values.yaml

Loading...
View Options

integration_tests/python_modules/dagster-k8s-test-infra/dagster_k8s_test_infra/helm.py

Loading...