Page MenuHomePhabricator

Add deployment docs for Airflow and Dask
ClosedPublic

Authored by natekupp on Jul 18 2019, 11:03 PM.

Details

Reviewers
alangenfeld
max
Group Reviewers
Restricted Project
Commits
R1:37ea6637d14d: Add deployment docs for Airflow and Dask
Test Plan

docs only

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

natekupp created this revision.Jul 18 2019, 11:03 PM
natekupp edited reviewers, added: Restricted Project; removed: max, schrockn.Jul 19 2019, 5:40 PM
alangenfeld added inline comments.
docs/sections/deploying/airflow.md
4

and works with all of your existing Airflow

this reads as a pretty bold statement of fact - i would consider

Dagster is designed for incremental adoption, and to work with all of your existing Airflow infrastructure.

10–12

this is confusing given the text below about containerized using a different operator

103–105

might be good to have some extra framing here - this limitation jumped out at me as off putting

alangenfeld requested changes to this revision.Jul 23 2019, 3:08 PM
This revision now requires changes to proceed.Jul 23 2019, 3:08 PM
natekupp updated this revision to Diff 3111.Jul 23 2019, 11:24 PM
natekupp marked 3 inline comments as done.

up

docs/sections/deploying/airflow.md
4

Great point, I like your version much better - thanks

10–12

I think I accidentally a word here - just going to drop this whole sentence.

103–105

yeah it's tough to frame this in a way that doesn't seem like asking for a lot of extra busywork... I took another cut at it, lmk what you think

max accepted this revision.Jul 24 2019, 6:11 PM
max added a subscriber: max.
max added inline comments.
docs/sections/deploying/airflow.md
29

i presume the k8s executor is sufficiently provisional that we dont need to mention it here also.. could say "or another distributed executor"

36

this is a heavy install

88

sweeps this

105

the local filesystem/in-memory will work only when the executors are running on the same node as the master, or when they share a filesystem

230

This landed in 1.10.3 -- we can update this to say "Airflow versions less than 1.10.3 are incompatible with py37"

alangenfeld accepted this revision.Jul 25 2019, 3:54 PM
This revision is now accepted and ready to land.Jul 25 2019, 3:54 PM
natekupp added inline comments.Jul 25 2019, 4:15 PM
docs/sections/deploying/airflow.md
29

yeah, also the k8s executor is brand new so I don't want to front-load mentioning it

natekupp marked 4 inline comments as done.Jul 25 2019, 4:17 PM
natekupp added inline comments.
docs/sections/deploying/airflow.md
105

added "on a single machine" here - maybe worth doing a future rev of this doc that pulls out the intermediate storage discussion to a separate section since it cross-cuts containerization modes.