Changeset View
Changeset View
Standalone View
Standalone View
docs/next/src/pages/deploying/k8s_part1.mdx
| Show First 20 Lines • Show All 57 Lines • ▼ Show 20 Lines | |||||
| helm install dagster dagster/dagster -f /path/to/values.yaml | helm install dagster dagster/dagster -f /path/to/values.yaml | ||||
| ``` | ``` | ||||
| _**Note:** In this example, Dagit runs with the user code image. In | _**Note:** In this example, Dagit runs with the user code image. In | ||||
| [Deploying on Kubernetes Part 2](https://docs.dagster.io/deploying/k8s_part2), we relax this constraint._ | [Deploying on Kubernetes Part 2](https://docs.dagster.io/deploying/k8s_part2), we relax this constraint._ | ||||
| ## System overview | ## System overview | ||||
| The Helm chart installs several different components, including Dagit. In this system, the <PyObject | The Helm chart installs several different components, including Dagit. | ||||
| module="dagster_celery_k8s" object="CeleryK8sRunLauncher" /> coupled with the corresponding | |||||
| `celery_k8s_job_executor` handles physical execution of Dagster pipelines via Celery-on-K8s. This | ### Celery K8s | ||||
| deployment aims to provide: | By default, we configure Dagster to deploy with Celery. | ||||
| In this system, the <PyObject module="dagster_celery_k8s" object="CeleryK8sRunLauncher" /> coupled | |||||
| with the corresponding <PyObject module="dagster_celery_k8s" object="celery_k8s_job_executor" /> | |||||
| handles physical execution of Dagster pipelines via Celery-on-K8s. This deployment aims to provide: | |||||
| - **User Code Separation:** In this system, the Celery workers can be deployed via fixed images | - **User Code Separation:** In this system, the Celery workers can be deployed via fixed images | ||||
| without user code, but rest of the system still uses the user code image. In the next part, we | without user code, but rest of the system still uses the user code image. In the next part, we | ||||
| walk through how user code can be deployed separately from all Dagster system components. | walk through how user code can be deployed separately from all Dagster system components. | ||||
| - **Priorities & Queueing:** We include Celery to take advantage of its support for task | - **Priorities & Queueing:** We include Celery to take advantage of its support for task | ||||
| priorities and queue widths. When using this system, you can annotate your solids with priority | priorities and queue widths. When using this system, you can annotate your solids with priority | ||||
| tags to prioritize certain solid executions, and with queue tags to manage parallelism and limit | tags to prioritize certain solid executions, and with queue tags to manage parallelism and limit | ||||
| the number of connections to resources. This works well but is not yet documented, so please | the number of connections to resources. This works well but is not yet documented, so please | ||||
| Show All 10 Lines | |||||
| run coordinator [Job](https://kubernetes.io/docs/concepts/workloads/controllers/job/) with the | run coordinator [Job](https://kubernetes.io/docs/concepts/workloads/controllers/job/) with the | ||||
| user code image specified in the `pipeline_run:` values YAML configuration (the run coordinator | user code image specified in the `pipeline_run:` values YAML configuration (the run coordinator | ||||
| Job will be named `dagster-run-<run_id>`, to make debugging in kubectl easier). The role of the run | Job will be named `dagster-run-<run_id>`, to make debugging in kubectl easier). The role of the run | ||||
| coordinator Job is to traverse the execution plan, and enqueue execution steps as Celery tasks. | coordinator Job is to traverse the execution plan, and enqueue execution steps as Celery tasks. | ||||
| Celery workers poll for new tasks, and for each step execution task that it fetches, a | Celery workers poll for new tasks, and for each step execution task that it fetches, a | ||||
| step execution Job is launched to execute that step. These step execution Jobs are also launched | step execution Job is launched to execute that step. These step execution Jobs are also launched | ||||
| using the `pipeline_run:` image, and each will be named `dagster-job-<hash>`. | using the `pipeline_run:` image, and each will be named `dagster-job-<hash>`. | ||||
| _**Note:** In the default Helm chart, we configure a <PyObject module="dagster_celery_k8s" object="CeleryK8sRunLauncher" />. | ### Celery-less K8s | ||||
| It is also possible to deploy with a <PyObject module="dagster_k8s" object="K8sRunLauncher" /> or a custom Run Launcher. | The published helm chart also supports a celery-less deployment. In this case, we configure the <PyObject | ||||
catherinewu: Proposing a slight rewording: `The published helm chart also supports a celery-less deployment. | |||||
| module="dagster_k8s" object="K8sRunLauncher" /> on the Dagster instance. This will launch each | |||||
| pipeline and its associated step executions in a single Kubernetes Job. We can use this option | |||||
| with the following command: | |||||
| ```shell | |||||
| helm install dagster dagster/dagster -f /path/to/values.yaml --set celery.enabled=false | |||||
Done Inline Actionsshould be updated given D4840 catherinewu: should be updated given D4840 | |||||
Done Inline Actionsupdated the docs in D4840 rexledesma: updated the docs in D4840 | |||||
| ``` | |||||
| _**Note:** It is also possible to deploy with a custom Run Launcher. | |||||
| If you have questions, please reach out to us on [Slack](https://dagster-slackin.herokuapp.com/) and we're happy to help!_ | If you have questions, please reach out to us on [Slack](https://dagster-slackin.herokuapp.com/) and we're happy to help!_ | ||||
| ## Helm chart | ## Helm chart | ||||
| As a full inventory, the Helm chart sets up the following components by default: | As a full inventory, the Helm chart sets up the following components by default: | ||||
| - Dagit running as a | - Dagit running as a | ||||
| [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) behind a | [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) behind a | ||||
| Show All 36 Lines | |||||
Proposing a slight rewording: The published helm chart also supports a celery-less deployment. In this case, we configure the <PyObject module="dagster_k8s" object="K8sRunLauncher" /> on the dagster instance, which will launch each pipeline (including all of its steps) in one Kubernetes Job. To use this option, <<insert steps here>>
We may want to mention that K8sRunLaunchers are compatible with the K8sScheduler and GRPC, but this is not the right place since those concepts are not yet introduced till the next part 2. On the other hand, maybe that's obvious to the reader.
I would also lean on the side of not specifying the compatible executors since this run launcher is compatible with all but the celery executors so it could be misleading?