Changeset View
Changeset View
Standalone View
Standalone View
docs/sections/intro_tutorial/hello_dag.rst
- This file was moved from docs/sections/learn/tutorial/hello_dag.rst.
Show All 10 Lines | |||||
Let's get serial | Let's get serial | ||||
^^^^^^^^^^^^^^^^ | ^^^^^^^^^^^^^^^^ | ||||
We'll add a second solid to the pipeline we worked with in the first section of the tutorial. | We'll add a second solid to the pipeline we worked with in the first section of the tutorial. | ||||
This new solid will consume the output of the first solid, which read the cereal dataset in from | This new solid will consume the output of the first solid, which read the cereal dataset in from | ||||
disk, and in turn will sort the list of cereals by their calorie content per serving. | disk, and in turn will sort the list of cereals by their calorie content per serving. | ||||
.. literalinclude:: ../../../../examples/dagster_examples/intro_tutorial/serial_pipeline.py | .. literalinclude:: ../../../examples/dagster_examples/intro_tutorial/serial_pipeline.py | ||||
:linenos: | :linenos: | ||||
:lines: 1-37 | :lines: 1-37 | ||||
:emphasize-lines: 15, 19, 37 | :emphasize-lines: 15, 19, 37 | ||||
:caption: serial_pipeline.py | :caption: serial_pipeline.py | ||||
You'll see that we've modified our existing ``load_cereals`` solid to return an output, in this | You'll see that we've modified our existing ``load_cereals`` solid to return an output, in this | ||||
case the list of dicts into which :py:class:``csv.DictReader <python:csv.DictReader>`` reads the | case the list of dicts into which :py:class:``csv.DictReader <python:csv.DictReader>`` reads the | ||||
cereals dataset. | cereals dataset. | ||||
Show All 18 Lines | |||||
A more complex DAG | A more complex DAG | ||||
^^^^^^^^^^^^^^^^^^ | ^^^^^^^^^^^^^^^^^^ | ||||
Solids don't need to be wired together serially. The output of one solid can be consumed by any | Solids don't need to be wired together serially. The output of one solid can be consumed by any | ||||
number of other solids, and the outputs of several different solids can be consumed by a single | number of other solids, and the outputs of several different solids can be consumed by a single | ||||
solid. | solid. | ||||
.. literalinclude:: ../../../../examples/dagster_examples/intro_tutorial/complex_pipeline.py | .. literalinclude:: ../../../examples/dagster_examples/intro_tutorial/complex_pipeline.py | ||||
:linenos: | :linenos: | ||||
:lines: 6-64 | :lines: 6-64 | ||||
:lineno-start: 6 | :lineno-start: 6 | ||||
:emphasize-lines: 55-59 | :emphasize-lines: 55-59 | ||||
:caption: complex_pipeline.py | :caption: complex_pipeline.py | ||||
First we introduce the intermediate variable ``cereals`` into our pipeline definition to | First we introduce the intermediate variable ``cereals`` into our pipeline definition to | ||||
represent the output of the ``load_cereals`` solid. Then we make both ``sort_by_calories`` and | represent the output of the ``load_cereals`` solid. Then we make both ``sort_by_calories`` and | ||||
Show All 13 Lines | |||||
execute not just in any order, but at the same time, since they don't depend on each other's | execute not just in any order, but at the same time, since they don't depend on each other's | ||||
outputs -- but both would still have to execute after ``load_cereals`` (because they depend on its | outputs -- but both would still have to execute after ``load_cereals`` (because they depend on its | ||||
output) and before ``display_results`` (because ``display_results`` depends on both of | output) and before ``display_results`` (because ``display_results`` depends on both of | ||||
their outputs). | their outputs). | ||||
We'll write a simple test for this pipeline showing how we can assert that all four of its solids | We'll write a simple test for this pipeline showing how we can assert that all four of its solids | ||||
executed successfully. | executed successfully. | ||||
.. literalinclude:: ../../../../examples/dagster_examples/intro_tutorial/complex_pipeline.py | .. literalinclude:: ../../../examples/dagster_examples/intro_tutorial/complex_pipeline.py | ||||
:linenos: | :linenos: | ||||
:lines: 72-77 | :lines: 72-77 | ||||
:lineno-start: 72 | :lineno-start: 72 | ||||
:caption: complex_pipeline.py | :caption: complex_pipeline.py |