Changeset View
Changeset View
Standalone View
Standalone View
docs/sections/intro_tutorial/intro_airflow.rst
- This file was moved from docs/sections/learn/tutorial/intro_airflow.rst.
Show All 24 Lines | |||||
You'll also need to make sure that the Dagster pipeline you want to run using Airflow is available | You'll also need to make sure that the Dagster pipeline you want to run using Airflow is available | ||||
in the Python environments in which your Airflow webserver and worker run. | in the Python environments in which your Airflow webserver and worker run. | ||||
Scaffolding your pipeline for Airflow | Scaffolding your pipeline for Airflow | ||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||||
We'll use our familiar, simple demo pipeline: | We'll use our familiar, simple demo pipeline: | ||||
.. literalinclude:: ../../../../examples/dagster_examples/intro_tutorial/airflow.py | .. literalinclude:: ../../../examples/dagster_examples/intro_tutorial/airflow.py | ||||
:linenos: | :linenos: | ||||
:caption: airflow.py | :caption: airflow.py | ||||
To compile this existing pipeline to Airflow, we'll use the ``dagster-airflow`` CLI tool. By | To compile this existing pipeline to Airflow, we'll use the ``dagster-airflow`` CLI tool. By | ||||
default, this tool will write the Airflow-compatible DAG scaffold out to ``$AIRFLOW_HOME/dags``. | default, this tool will write the Airflow-compatible DAG scaffold out to ``$AIRFLOW_HOME/dags``. | ||||
.. code-block:: shell | .. code-block:: shell | ||||
$ dagster-airflow scaffold \ | $ dagster-airflow scaffold \ | ||||
--module-name dagster_examples.intro_tutorial.airflow \ | --module-name dagster_examples.intro_tutorial.airflow \ | ||||
--pipeline-name hello_cereal_pipeline | --pipeline-name hello_cereal_pipeline | ||||
Wrote DAG scaffold to file: $AIRFLOW_HOME/dags/hello_cereal_pipeline.py | Wrote DAG scaffold to file: $AIRFLOW_HOME/dags/hello_cereal_pipeline.py | ||||
Let's take a look at the generated file: | Let's take a look at the generated file: | ||||
.. literalinclude:: ../../../../examples/dagster_examples/intro_tutorial/hello_cereal_pipeline.py | .. literalinclude:: ../../../examples/dagster_examples/intro_tutorial/hello_cereal_pipeline.py | ||||
:linenos: | :linenos: | ||||
:caption: hello_cereal_pipeline.py | :caption: hello_cereal_pipeline.py | ||||
This is a fairly straightforward file with four parts. | This is a fairly straightforward file with four parts. | ||||
First, we import the basic prerequisites to define our DAG (and also make sure that the string | First, we import the basic prerequisites to define our DAG (and also make sure that the string | ||||
"DAG" appears in the file, so that the Airflow webserver will detect it). | "DAG" appears in the file, so that the Airflow webserver will detect it). | ||||
▲ Show 20 Lines • Show All 52 Lines • Show Last 20 Lines |