Differential D8684 Diff 40976 docs/content/deployment/guides/celery.mdx

Changeset View

Standalone View

docs/content/deployment/guides/celery.mdx

	Show All 14 Lines

	The dagster-celery executor compiles a pipeline definition and its associated configuration into a concrete execution plan, and then submits each execution step to the broker as a separate Celery task. The dagster-celery workers then pick up tasks from the queues to which they are subscribed, according to the priorities assigned to each task, and execute the steps to which the tasks correspond.			The dagster-celery executor compiles a pipeline definition and its associated configuration into a concrete execution plan, and then submits each execution step to the broker as a separate Celery task. The dagster-celery workers then pick up tasks from the queues to which they are subscribed, according to the priorities assigned to each task, and execute the steps to which the tasks correspond.

	## Quick start			## Quick start

	Let's construct a very parallel toy pipeline and then execute it using the Celery executor.			Let's construct a very parallel toy pipeline and then execute it using the Celery executor.

	```python file=/deploying/celery_pipeline.py			```python file=/deploying/celery_pipeline.py
	from dagster import ModeDefinition, default_executors, fs_io_manager, pipeline, solid			from dagster import (
				ModeDefinition,
				default_executors,
				fs_io_manager,
				pipeline,
				solid,
				)
	from dagster_celery import celery_executor			from dagster_celery import celery_executor

	celery_mode_defs = [			celery_mode_defs = [
	ModeDefinition(			ModeDefinition(
	resource_defs={"io_manager": fs_io_manager},			resource_defs={"io_manager": fs_io_manager},
	executor_defs=default_executors + [celery_executor],			executor_defs=default_executors + [celery_executor],
	)			)
	]			]
	▲ Show 20 Lines • Show All 129 Lines • Show Last 20 Lines