Differential D1976 Diff 9519 docs/sections/intro_tutorial/hello_dag.rst

Changeset View

Standalone View

docs/sections/intro_tutorial/hello_dag.rst

This file was moved from docs/sections/learn/tutorial/hello_dag.rst.

	Show All 10 Lines
	Let's get serial			Let's get serial
	^^^^^^^^^^^^^^^^			^^^^^^^^^^^^^^^^

	We'll add a second solid to the pipeline we worked with in the first section of the tutorial.			We'll add a second solid to the pipeline we worked with in the first section of the tutorial.

	This new solid will consume the output of the first solid, which read the cereal dataset in from			This new solid will consume the output of the first solid, which read the cereal dataset in from
	disk, and in turn will sort the list of cereals by their calorie content per serving.			disk, and in turn will sort the list of cereals by their calorie content per serving.

	.. literalinclude:: ../../../../examples/dagster_examples/intro_tutorial/serial_pipeline.py			.. literalinclude:: ../../../examples/dagster_examples/intro_tutorial/serial_pipeline.py
	:linenos:			:linenos:
	:lines: 1-37			:lines: 1-37
	:emphasize-lines: 15, 19, 37			:emphasize-lines: 15, 19, 37
	:caption: serial_pipeline.py			:caption: serial_pipeline.py

	You'll see that we've modified our existing ``load_cereals`` solid to return an output, in this			You'll see that we've modified our existing ``load_cereals`` solid to return an output, in this
	case the list of dicts into which :py:class:``csv.DictReader <python:csv.DictReader>`` reads the			case the list of dicts into which :py:class:``csv.DictReader <python:csv.DictReader>`` reads the
	cereals dataset.			cereals dataset.
	Show All 18 Lines

	A more complex DAG			A more complex DAG
	^^^^^^^^^^^^^^^^^^			^^^^^^^^^^^^^^^^^^

	Solids don't need to be wired together serially. The output of one solid can be consumed by any			Solids don't need to be wired together serially. The output of one solid can be consumed by any
	number of other solids, and the outputs of several different solids can be consumed by a single			number of other solids, and the outputs of several different solids can be consumed by a single
	solid.			solid.

	.. literalinclude:: ../../../../examples/dagster_examples/intro_tutorial/complex_pipeline.py			.. literalinclude:: ../../../examples/dagster_examples/intro_tutorial/complex_pipeline.py
	:linenos:			:linenos:
	:lines: 6-64			:lines: 6-64
	:lineno-start: 6			:lineno-start: 6
	:emphasize-lines: 55-59			:emphasize-lines: 55-59
	:caption: complex_pipeline.py			:caption: complex_pipeline.py

	First we introduce the intermediate variable ``cereals`` into our pipeline definition to			First we introduce the intermediate variable ``cereals`` into our pipeline definition to
	represent the output of the ``load_cereals`` solid. Then we make both ``sort_by_calories`` and			represent the output of the ``load_cereals`` solid. Then we make both ``sort_by_calories`` and
	Show All 13 Lines
	execute not just in any order, but at the same time, since they don't depend on each other's			execute not just in any order, but at the same time, since they don't depend on each other's
	outputs -- but both would still have to execute after ``load_cereals`` (because they depend on its			outputs -- but both would still have to execute after ``load_cereals`` (because they depend on its
	output) and before ``display_results`` (because ``display_results`` depends on both of			output) and before ``display_results`` (because ``display_results`` depends on both of
	their outputs).			their outputs).

	We'll write a simple test for this pipeline showing how we can assert that all four of its solids			We'll write a simple test for this pipeline showing how we can assert that all four of its solids
	executed successfully.			executed successfully.

	.. literalinclude:: ../../../../examples/dagster_examples/intro_tutorial/complex_pipeline.py			.. literalinclude:: ../../../examples/dagster_examples/intro_tutorial/complex_pipeline.py
	:linenos:			:linenos:
	:lines: 72-77			:lines: 72-77
	:lineno-start: 72			:lineno-start: 72
	:caption: complex_pipeline.py			:caption: complex_pipeline.py