Page MenuHomePhabricator

(helm) set up shared env vars
ClosedPublic

Authored by nate on Wed, May 13, 3:19 AM.

Details

Summary

This diff consolidates shared env vars in one place to avoid drift between dagit, celery, etc.

Also, by converting the run master configs to environment variables, it is easier to keep configuration in sync across the run master and step execution pods. Pipelines can now be executed with the config:

execution:
  celery-k8s:
    config:
      backend:
        env: DAGSTER_K8S_CELERY_BACKEND
      broker:
        env: DAGSTER_K8S_CELERY_BACKEND
      job_image:
        env: DAGSTER_K8S_PIPELINE_RUN_IMAGE
      job_namespace:  
        env: DAGSTER_K8S_PIPELINE_RUN_NAMESPACE
      instance_config_map:
        env: DAGSTER_K8S_INSTANCE_CONFIG_MAP
      image_pull_policy: "Always"
      postgres_password_secret:
        env: DAGSTER_K8S_PG_PASSWORD_SECRET
      env_config_maps: 
        - env: DAGSTER_K8S_PIPELINE_RUN_ENV_CONFIGMAP

And this will pick up the appropriate step execution configs from the container environment.

Test Plan

buildkite

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

nate created this revision.Wed, May 13, 3:19 AM
nate edited the summary of this revision. (Show Details)Wed, May 13, 3:22 AM
nate added reviewers: catherinewu, max, alangenfeld.
nate requested review of this revision.Wed, May 13, 3:35 AM
nate updated this revision to Diff 13839.Wed, May 13, 4:17 AM
nate edited the summary of this revision. (Show Details)

up

max accepted this revision.Wed, May 13, 4:35 PM
This revision is now accepted and ready to land.Wed, May 13, 4:35 PM
This revision was automatically updated to reflect the committed changes.