Page MenuHomeElementl

RFC: Remove functionality from k8s run launchers to reload the code pointer and image name before launching
ClosedPublic

Authored by dgibson on Feb 15 2021, 10:48 PM.

Details

Summary

Way back in 0.9.0 when we were first getting the k8s user code deployments working end-to-end, we made the k8s and celery k8s run launcher reload the pipeline python origin from the gRPC server during the run launch. My recollection is that this was to support the use case of our main external user at the time, who wanted to be able to ensure that they were always running from the latest code.

However, since this feature was added, we've added other ways to keep dagit in sync with the latest version of code - primarily by improving the 'reload repository location' feature. In particular, dagit now automatically monitors the state of any user code deployments and prompts you to refresh them when they are updated, thanks to sashank. I think we should now consider removing this auto-reload-during launch functionality (we could also make it a setting on the k8s run launcher). That way, we can ensure that the code that you validated your config against is actually the config being launched, and ensure that the same metadata is being used throughout the system. Future features may also increase the latency associated with hitting the gRPC server, making it useful to reduce the number of gRPC API calls that we make.

We could also consider making this a setting on the run launcher if we feel that this functionality is still useful to current users.

Test Plan

Integration

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Macro chefkiss:

python_modules/libraries/dagster-celery-k8s/dagster_celery_k8s/launcher.py
168–169

can delete "job_image = None" and "pipeline_origin = None"

206

e

python_modules/libraries/dagster-k8s/dagster_k8s/launcher.py
232–233

can delete "pipeline_origin = None" and " job_config = None"

This revision is now accepted and ready to land.Feb 17 2021, 7:32 PM
python_modules/libraries/dagster-celery-k8s/dagster_celery_k8s/launcher.py
206

^ignore