Differential D1976 Diff 9519 docs/sections/intro_tutorial/intro_airflow.rst

Changeset View

Standalone View

docs/sections/intro_tutorial/intro_airflow.rst

This file was moved from docs/sections/learn/tutorial/intro_airflow.rst.

	Show All 24 Lines

	You'll also need to make sure that the Dagster pipeline you want to run using Airflow is available			You'll also need to make sure that the Dagster pipeline you want to run using Airflow is available
	in the Python environments in which your Airflow webserver and worker run.			in the Python environments in which your Airflow webserver and worker run.

	Scaffolding your pipeline for Airflow			Scaffolding your pipeline for Airflow
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^			^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	We'll use our familiar, simple demo pipeline:			We'll use our familiar, simple demo pipeline:

	.. literalinclude:: ../../../../examples/dagster_examples/intro_tutorial/airflow.py			.. literalinclude:: ../../../examples/dagster_examples/intro_tutorial/airflow.py
	:linenos:			:linenos:
	:caption: airflow.py			:caption: airflow.py

	To compile this existing pipeline to Airflow, we'll use the ``dagster-airflow`` CLI tool. By			To compile this existing pipeline to Airflow, we'll use the ``dagster-airflow`` CLI tool. By
	default, this tool will write the Airflow-compatible DAG scaffold out to ``$AIRFLOW_HOME/dags``.			default, this tool will write the Airflow-compatible DAG scaffold out to ``$AIRFLOW_HOME/dags``.

	.. code-block:: shell			.. code-block:: shell

	$ dagster-airflow scaffold \			$ dagster-airflow scaffold \
	--module-name dagster_examples.intro_tutorial.airflow \			--module-name dagster_examples.intro_tutorial.airflow \
	--pipeline-name hello_cereal_pipeline			--pipeline-name hello_cereal_pipeline
	Wrote DAG scaffold to file: $AIRFLOW_HOME/dags/hello_cereal_pipeline.py			Wrote DAG scaffold to file: $AIRFLOW_HOME/dags/hello_cereal_pipeline.py

	Let's take a look at the generated file:			Let's take a look at the generated file:

	.. literalinclude:: ../../../../examples/dagster_examples/intro_tutorial/hello_cereal_pipeline.py			.. literalinclude:: ../../../examples/dagster_examples/intro_tutorial/hello_cereal_pipeline.py
	:linenos:			:linenos:
	:caption: hello_cereal_pipeline.py			:caption: hello_cereal_pipeline.py

	This is a fairly straightforward file with four parts.			This is a fairly straightforward file with four parts.

	First, we import the basic prerequisites to define our DAG (and also make sure that the string			First, we import the basic prerequisites to define our DAG (and also make sure that the string
	"DAG" appears in the file, so that the Airflow webserver will detect it).			"DAG" appears in the file, so that the Airflow webserver will detect it).

	▲ Show 20 Lines • Show All 52 Lines • Show Last 20 Lines