Page MenuHomePhabricator

[celery-k8s] retry tweaks
ClosedPublic

Authored by alangenfeld on Fri, Oct 2, 3:11 PM.

Details

Summary
  • sync retries with wait_time_between_attempts
  • make max_retries interpret the initial attempt not as a retry
  • dont retry on 429, do retry on 503
Test Plan

bk

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Fri, Oct 2, 3:50 PM
Harbormaster failed remote builds in B19039: Diff 23124!
Harbormaster returned this revision to the author for changes because remote builds failed.Fri, Oct 2, 4:17 PM
Harbormaster failed remote builds in B19045: Diff 23132!
Harbormaster returned this revision to the author for changes because remote builds failed.Fri, Oct 2, 5:03 PM
Harbormaster failed remote builds in B19054: Diff 23141!
Harbormaster returned this revision to the author for changes because remote builds failed.Fri, Oct 2, 5:42 PM
Harbormaster failed remote builds in B19067: Diff 23157!

I think this makes sense, especially since these gateway timeout errors are not going to resolve themselves within 10 seconds.

python_modules/libraries/dagster-k8s/dagster_k8s_tests/unit_tests/test_client.py
146–150

Do you know how to check if the underlying cause is correct? I couldn't figure it out when I was writing the test. Might be good to add here.

python_modules/libraries/dagster-k8s/dagster_k8s_tests/unit_tests/test_client.py
146–150

I figured DagsterK8sAPIRetryLimitExceeded was sufficient

alangenfeld edited the test plan for this revision. (Show Details)
alangenfeld edited the summary of this revision. (Show Details)
Harbormaster returned this revision to the author for changes because remote builds failed.Mon, Oct 5, 5:09 PM
Harbormaster failed remote builds in B19134: Diff 23250!
Harbormaster returned this revision to the author for changes because remote builds failed.Mon, Oct 5, 6:43 PM
Harbormaster failed remote builds in B19136: Diff 23254!
alangenfeld retitled this revision from [celery-k8s] up timeouts to [celery-k8s] retry tweaks.Tue, Oct 6, 7:24 PM
alangenfeld edited the summary of this revision. (Show Details)
alangenfeld retitled this revision from [celery-k8s] retry tweaks to [celery-k8s] up timeouts.
alangenfeld edited the summary of this revision. (Show Details)

up

This revision is now accepted and ready to land.Tue, Oct 6, 9:19 PM
alangenfeld retitled this revision from [celery-k8s] up timeouts to [celery-k8s] retry tweaks.Tue, Oct 6, 9:24 PM
alangenfeld edited the summary of this revision. (Show Details)
This revision was automatically updated to reflect the committed changes.