Details

Reviewers

Commits

R1:c91e921f0a73: [1/n] Add helm schema validation for Dagit configuration

Summary

This sets up the boilerplate for validating the values.yaml file using a values.schema.json file. The schema file is created through pydantic's DSL. I've also added a cli tool to update the helm repo's schema. The workflow for updating the schema should look something like this:

Make changes to pydantic models
Show changes using dagster-helm schema | git diff --no-index --exit-code -- helm/dagster/values.schema.json -
If the changes are good, dagster-helm schema --command=apply to update the current values.schema.json.

As an example, we add schema validation for the helm values that configure the Dagit webserver.

Test Plan

helm lint helm/dagster -f helm/dagster/values.yaml, bk

Diff Detail

Repository

R1 dagster

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

• rexledesma created this revision.Oct 24 2020, 12:58 AM

• rexledesma added inline comments.Oct 24 2020, 1:11 AM

helm/dagster/values.schema.json
88	Do we advertise a specific Kubernetes that we support? I used `1.15.0` since our python version of `11.0.0` k8s has feature parity with it.
88	Also, these definitions are a god send. Was in the middle of implementing some of them by hand when I realized that there's probably some community contributed source for these.

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 24 2020, 1:19 AM

Harbormaster failed remote builds in B20118: Diff 24415!

• rexledesma added inline comments.Oct 24 2020, 1:28 AM

helm/dagster/values.schema.json
41	These properties (`env`, `env_config_maps`, `env_secrets`) should really follow the schema of the configs they model in kubernetes (e.g. `env_config_maps` -> https://kubernetesjsonschema.dev/v1.15.0/_definitions.json#/definitions/io.k8s.api.core.v1.EnvFromSource). Then, there will be no need to wrangle these values into their required form - they can simply be interpolated with a simple `nindent`.

rebase

Harbormaster completed remote builds in B20170: Diff 24483.Oct 26 2020, 8:36 PM

• rexledesma requested review of this revision.Oct 26 2020, 8:36 PM

TEST PLAN
helm lint helm/dagster -f helm/dagster/values.yaml

does helm only use these values to power lint or is it a deeper integration?

helm/dagster/values.schema.json
2	I feel like this should likely be the output of a program instead of something we manually manage - doing that will make it much easier to expand to all the other parts of the schema
41	will these changes be breaking changes? if so queue them up for `0.10.0`
88	im not sure which version we should pin to - but moving this to be the output of a program should make it much easier to manage / change that out
helm/dagster/values.yaml
314	🥴

• rexledesma marked 2 inline comments as done.Oct 27 2020, 7:13 PM

• rexledesma added inline comments.

helm/dagster/values.schema.json
2	Good idea - here's a nice DSL that we can use: https://pydantic-docs.helpmanual.io/usage/schema/ - I'll put some generator under `python_modules/automation/automation/helm`

• rexledesma removed reviewers: • catherinewu, • sashank, johann, alangenfeld, • bob.Oct 27 2020, 7:13 PM

Adds dagster-helm cli to programatically generate schema file
Add buildkite from D4897

Harbormaster failed remote builds in B20280: Diff 24606!Oct 27 2020, 10:43 PM

ignore lint issues - https://github.com/samuelcolvin/pydantic/issues/1961

• rexledesma retitled this revision from Add helm schema validation for Dagit configuration to [1/n] Add helm schema validation for Dagit configuration.Oct 27 2020, 10:57 PM

• rexledesma edited the summary of this revision. (Show Details)

• rexledesma edited the test plan for this revision. (Show Details)

black

Harbormaster failed remote builds in B20282: Diff 24608!Oct 27 2020, 11:01 PM

Harbormaster completed remote builds in B20284: Diff 24611.Oct 27 2020, 11:23 PM

• rexledesma added a child revision: D4920: Update Scheduler helm schema to use enum types.Oct 28 2020, 12:20 AM

rebase

Harbormaster completed remote builds in B20289: Diff 24618.Oct 28 2020, 12:59 AM

Harbormaster completed remote builds in B20292: Diff 24621.Oct 28 2020, 1:13 AM

• rexledesma edited the summary of this revision. (Show Details)Oct 28 2020, 1:30 AM

• rexledesma added a reviewer: alangenfeld.

@alangenfeld Thanks for the suggestion - I added a cli tool to generate the schema file. Also, the buildkite step that was added will ensure the following things:

dagster-helm schema --command=apply was checked into git if there was change to the pydantic schema
Our values.yaml is well formed according to our generated schema. This makes use of a deeper integration in helm, so that we can apply the schema.

• rexledesma marked an inline comment as done.Oct 28 2020, 1:38 AM

• rexledesma added inline comments.

helm/dagster/values.schema.json
41	I'll refactor these properties as camelcase + use the new $ref schema in a separate diff.

redfordnod

.buildkite/pipeline.py
709	`\|` here feels odd to me - would this be clearer if it was just two separate commands? a blanket `git diff --exit-code` also is a bit more future proof incase we start generating other outputs
python_modules/automation/automation/helm/schema/dagit.py
20–31	ya this is really nice
python_modules/automation/automation/helm/schema/kubernetes.py
33	ah inner class shenanigans

This revision is now accepted and ready to land.Oct 28 2020, 3:45 PM

Harbormaster failed remote builds in B20340: Diff 24674!Oct 28 2020, 6:47 PM

Harbormaster completed remote builds in B20341: Diff 24675.Oct 28 2020, 6:50 PM

Closed by commit R1:c91e921f0a73: [1/n] Add helm schema validation for Dagit configuration (authored by • rexledesma). · Explain WhyOct 28 2020, 6:51 PM

This revision was automatically updated to reflect the committed changes.

• rexledesma added a commit: R1:c91e921f0a73: [1/n] Add helm schema validation for Dagit configuration.

[1/n] Add helm schema validation for Dagit configuration
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 24678

.buildkite/pipeline.py

helm/dagster/values.schema.json

helm/dagster/values.yaml

python_modules/automation/automation/helm/init.py

python_modules/automation/automation/helm/cli.py

python_modules/automation/automation/helm/schema/init.py

python_modules/automation/automation/helm/schema/dagit.py

python_modules/automation/automation/helm/schema/kubernetes.py

python_modules/automation/automation/helm/schema/utils.py

python_modules/automation/automation/helm/schema/values.py

python_modules/automation/setup.py

[1/n] Add helm schema validation for Dagit configurationClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 24678

.buildkite/pipeline.py

helm/dagster/values.schema.json

helm/dagster/values.yaml

python_modules/automation/automation/helm/__init__.py

python_modules/automation/automation/helm/cli.py

python_modules/automation/automation/helm/schema/__init__.py

python_modules/automation/automation/helm/schema/dagit.py

python_modules/automation/automation/helm/schema/kubernetes.py

python_modules/automation/automation/helm/schema/utils.py

python_modules/automation/automation/helm/schema/values.py

python_modules/automation/setup.py

[1/n] Add helm schema validation for Dagit configuration
ClosedPublic
Actions

Revision Contents
Changeset List

python_modules/automation/automation/helm/init.py

python_modules/automation/automation/helm/schema/init.py