Page MenuHomePhabricator

Dask and docker-compose
ClosedPublic

Authored by natekupp on May 29 2019, 5:21 AM.

Details

Test Plan

docker-compose

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

natekupp created this revision.May 29 2019, 5:21 AM
natekupp updated this revision to Diff 1164.May 29 2019, 5:32 AM

fix failure

natekupp updated this revision to Diff 1170.May 29 2019, 4:02 PM

add dagster-aws

natekupp updated this revision to Diff 1176.May 29 2019, 4:37 PM

dask address

natekupp updated this revision to Diff 1196.May 29 2019, 6:21 PM

it actually works

natekupp edited the test plan for this revision. (Show Details)May 29 2019, 6:22 PM
natekupp added reviewers: schrockn, max, alangenfeld.
natekupp updated this revision to Diff 1199.May 29 2019, 6:26 PM

Try a different method of getting container ID

max added inline comments.May 29 2019, 6:34 PM
.buildkite/pipeline.py
265

ok -- i think we could also run the tox tests from another container in the same docker-compose, which would simplify the networking at the expense of having to ship the coverage files back to the host

python_modules/dagster-dask/dagster_dask_tests/dask-docker/Dockerfile
6

can this go in a requirements.txt

alangenfeld added inline comments.May 29 2019, 6:40 PM
.buildkite/pipeline.py
256

pushd

257–258

maybe move to the integration image?

262–264

pretty sure we can go back to --remove-orphans instead of the stop and rm - the end result was the same

265–273

scaredbaby

274

popd

python_modules/dagster-dask/dagster_dask_tests/dask-docker/build.sh
3–7

prefer

ROOT=$(git rev-parse --show-toplevel)
pushd $ROOT/python_modules/dagster-dask/dagster_dask_tests/dask-docker/

see

https://dagster.phacility.com/source/dagster/browse/master/examples/build_airline_demo_image.sh

natekupp updated this revision to Diff 1206.May 29 2019, 7:35 PM

attempt escaping

natekupp updated this revision to Diff 1207.May 29 2019, 7:38 PM

escape only $

natekupp updated this revision to Diff 1208.May 29 2019, 7:42 PM

one more

natekupp updated this revision to Diff 1209.May 29 2019, 8:05 PM
natekupp marked 4 inline comments as done.

comments

natekupp added inline comments.May 29 2019, 8:07 PM
.buildkite/pipeline.py
256

done

python_modules/dagster-dask/dagster_dask_tests/dask-docker/build.sh
3–7

cool, done

natekupp updated this revision to Diff 1210.May 29 2019, 8:12 PM

fix and remove-orphans

natekupp marked an inline comment as done.May 29 2019, 8:16 PM
natekupp added inline comments.
.buildkite/pipeline.py
262–264

done

natekupp updated this revision to Diff 1212.May 29 2019, 8:21 PM
natekupp marked an inline comment as done.

attempt to install bokeh from install_requires

natekupp updated this revision to Diff 1223.May 29 2019, 10:10 PM
natekupp marked 2 inline comments as done.

comments

natekupp added inline comments.May 29 2019, 10:11 PM
.buildkite/pipeline.py
257–258

done

265

unless you feel strongly about this one, I'm fine leaving this as an exercise for a future reader of this code (probably me)

python_modules/dagster-dask/dagster_dask_tests/dask-docker/Dockerfile
6

yep, done

natekupp marked an inline comment as done.May 29 2019, 10:11 PM
natekupp updated this revision to Diff 1227.May 29 2019, 10:29 PM

Add retries and timeout to dask buildkite job

natekupp updated this revision to Diff 1228.May 29 2019, 10:53 PM

rebase and hope the build passes

natekupp updated this revision to Diff 1230.May 29 2019, 10:59 PM

try to address airline-demo build failure

max accepted this revision.May 29 2019, 11:57 PM
This revision is now accepted and ready to land.May 29 2019, 11:57 PM
This revision was automatically updated to reflect the committed changes.