HomePhabricator

Add k8s-dagit-example image to fix deployment docs

Description

Add k8s-dagit-example image to fix deployment docs

Summary:
Helm install out of the box is not working, since no workspace has been configured in the default k8s-dagit image.
This makes sense because it contains no pipeline code. However, it prevents dagit from loading since
it cannot find a workspace. k8s-dagit is like this since this was intended for user code deployments.

We introduce another image k8s-dagit-example that has both user code and a dagit dependency. By default, since
user code deployments are not enabled, this will allow users to helm install and provision a working dagit instance.

Test Plan:
Manually running pipeline after helm install test /users/rexledesma/elementl/dagster/helm/dagster -f helm-overrides.yaml.

  1. Out of the box, celery enabled
# helm-overrides.yaml

dagit:
  image:
    repository: <eks_cluster>/k8s-dagit-example
    tag: rexledesma-test

and the example preset run config (after configuring s3) runs the pipeline to success.

  1. Out of the box, celery disabled, using the k8s run launcher
# helm-overrides.yaml

celery:
  enabled: false

k8sRunLauncher:
  enabled: true
  image:
    repository:  <eks_cluster>/k8s-dagit-example
    tag: rexledesma-test
    pullPolicy: Always

dagit:
  image:
    repository:  <eks_cluster>/k8s-dagit-example
    tag: rexledesma-test

with run config

execution:
  multiprocess:

intermediate_storage:
  filesystem:
    config:
      base_dir: "/"

solids:
  multiply_the_word:
    config:
      factor: 2
    inputs:
      word: test

Reviewers: catherinewu, alangenfeld, johann, bob

Reviewed By: alangenfeld

Differential Revision: https://dagster.phacility.com/D5094

Details