Page MenuHomePhabricator

Add k8s-dagit-example image to fix deployment docs
ClosedPublic

Authored by rexledesma on Wed, Nov 11, 3:23 PM.

Details

Summary

Helm install out of the box is not working, since no workspace has been configured in the default k8s-dagit image.
This makes sense because it contains no pipeline code. However, it prevents dagit from loading since
it cannot find a workspace. k8s-dagit is like this since this was intended for user code deployments.

We introduce another image k8s-dagit-example that has both user code and a dagit dependency. By default, since
user code deployments are not enabled, this will allow users to helm install and provision a working dagit instance.

Test Plan

Manually running pipeline after helm install test /users/rexledesma/elementl/dagster/helm/dagster -f helm-overrides.yaml.

  1. Out of the box, celery enabled
# helm-overrides.yaml

dagit:
  image:
    repository: <eks_cluster>/k8s-dagit-example
    tag: rexledesma-test

and the example preset run config (after configuring s3) runs the pipeline to success.

  1. Out of the box, celery disabled, using the k8s run launcher
# helm-overrides.yaml

celery:
  enabled: false

k8sRunLauncher:
  enabled: true
  image:
    repository:  <eks_cluster>/k8s-dagit-example
    tag: rexledesma-test
    pullPolicy: Always

dagit:
  image:
    repository:  <eks_cluster>/k8s-dagit-example
    tag: rexledesma-test

with run config

execution:
  multiprocess:

intermediate_storage:
  filesystem:
    config:
      base_dir: "/"
      
solids:
  multiply_the_word:
    config:
      factor: 2
    inputs:
      word: test

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Helm install out of the box is not working

would it be better to just have the default dagit image have a workspace.yaml in the workdir?

the --empty-workspace flags forces an empty workspace even if there is a workspace.yaml in the cwd, so I don't think this is right - a more thorough test should confirm this

to your queue for ^

This revision now requires changes to proceed.Wed, Nov 11, 3:43 PM

we could set userDeployments.enabled to true instead (also a breaking change for users who depend on the default). we probably publish pipelines in the corresponding image: https://hub.docker.com/r/dagster/k8s-example

another possibility is to update the helm command in the docs to set userDeployments.enabled to true?

the --empty-workspace flags forces an empty workspace even if there is a workspace.yaml in the cwd, so I don't think this is right - a more thorough test should confirm this

@alangenfeld Looks like you're right. However, we can't add an empty workspace since that fails validation. I think the move is to have image dedicated for dagit running with user code (which I've added here).

The naming is confusing for these images. We should rename them as we do a pass through on our tutorial.

k8s-dagit (rename as just dagit), this should simply run dagit for user code deployments
k8s-dagit-example (rename as just dagit-example), this should have both dagit and user code
k8s-example (rename as just example_repo) , this should have just user code

To restructure the tutorial, we can go around with using k8s-dagit-example until we introduce user code separation later.

helm/dagster/values.yaml
20

this should actually be updated to "dagster/k8s-dagit-example", but not sure if we can update this in this diff since release for these images only occurs on Th

python_modules/automation/automation/docker/images/k8s-dagit-example/Dockerfile
12

need to pin these to dagster version

TEST PLAN
helm install

I assume you actually did more than this so update title, test plan, and summary before landing

make sure to follow the release today (if you land this today) to ensure the example gets built

python_modules/automation/automation/docker/image_defs.py
113

probably a follow-up diff, but we should include gcp as well and make sure the example can work on both

139

what do we need to update to ensure this gets released - just the quip ?

This revision is now accepted and ready to land.Thu, Nov 12, 6:10 PM
rexledesma retitled this revision from Allow empty workspace when starting up dagit in helm to Add k8s-dagit-example image to fix deployment docs.Thu, Nov 12, 8:28 PM
rexledesma edited the summary of this revision. (Show Details)
rexledesma edited the test plan for this revision. (Show Details)
rexledesma edited the test plan for this revision. (Show Details)