HomePhabricator

Dagster + Helm/Kubernetes

Authored by alangenfeld on Dec 6 2019, 5:15 PM.

Description

Dagster + Helm/Kubernetes

Summary:
This is the first cut at a Dagster Helm chart which installs Dagit and PG on a running Kubernetes cluster; credit to @alangenfeld for the majority of the templating work in Helm here.

Along with the Dagit instance, this diff also introduces a k8s run launcher which dynamically constructs k8s job objects and submits them for execution.

This diff also sets up testing so that we can run k8s tests in kind, building on the prior work with the k8s pod operator in dagster-airflow (see D1429).

Follow-ups:

  • We should think about a job GC strategy; we may want to remove jobs after they've completed and we've retrieved logs.
  • Wire up run launchers to dagit
  • This implements submission of pipeline runs as k8s jobs, we should also provide task-level execution

Test Plan: includes unit test which tests jobs creation and submission/execution

Reviewers: alangenfeld, schrockn

Reviewed By: alangenfeld

Subscribers: alangenfeld

Differential Revision: https://dagster.phacility.com/D1666

Details

Committed
nateDec 21 2019, 1:29 AM
Reviewer
alangenfeld
Differential Revision
D1666: Dagster + Helm/Kubernetes
Parents
R1:3d2e368dc949: add validation check to remote run launcher
Branches
Unknown
Tags
Unknown