Page MenuHomeElementl

Create a new k8s client on each run
ClosedPublic

Authored by dgibson on May 27 2021, 2:47 PM.
Tags
None
Referenced Files
Unknown Object (File)
Wed, May 24, 11:09 AM
Unknown Object (File)
Sun, May 14, 2:17 AM
Unknown Object (File)
Apr 7 2023, 10:32 AM
Unknown Object (File)
Mar 31 2023, 12:58 PM
Unknown Object (File)
Mar 29 2023, 10:38 PM
Unknown Object (File)
Mar 19 2023, 6:22 AM
Unknown Object (File)
Mar 16 2023, 7:01 PM
Unknown Object (File)
Mar 15 2023, 6:17 PM
Subscribers
None

Details

Summary

Users are reporting issues with stale k8s clients causing issues in the k8s run launcher (most troublingly, a hang on AKS when you try to launch a run) . We are already creating a client for each run in the celery k8s executor, and the current caching behavior seems to be mainly for test mocking purposes. Instead, create a new client on each run

Test Plan

Integration (then launch a run in the demo tier as a sanity check)

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

airflow tests failing in master

This revision is now accepted and ready to land.May 27 2021, 3:06 PM

nit: it appears the k8s launcher doesn't use _core_api?

python_modules/libraries/dagster-k8s/dagster_k8s/launcher.py
145

Else statement is wrong?

oof good catch thank you, will be sure to test manually too as this path likely doesn't have great coverage (since it's mocked out)

This revision was landed with ongoing or failed builds.May 27 2021, 4:40 PM
This revision was automatically updated to reflect the committed changes.