Page MenuHomePhabricator

Update integration Dockerfile to support building public Dagster images
ClosedPublic

Authored by natekupp on Thu, Sep 5, 5:38 AM.

Details

Summary

This cleans up and reorganizes our buildkite Dockerfile, and introduces a multi-stage build where we also build a dagster/dagster image.

I also added a Makefile to automate these builds, and included a Makefile target & script to generate the snapshot-reqs files which were previously manually generated.

Test Plan

built locally; tested examples in integration image

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /Users/nate/src/dagster:/workdir -it dagster/buildkite-integration:py3.7.3-v5 /bin/bash
pushd workdir/examples
tox -e py37

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

natekupp created this revision.Thu, Sep 5, 5:38 AM
natekupp added inline comments.Thu, Sep 5, 5:42 AM
.buildkite/images/docker/snapshot-reqs-2.txt
55

well I have no idea why this is here

Harbormaster completed remote builds in B3409: Diff 4258.
natekupp edited the summary of this revision. (Show Details)Thu, Sep 5, 5:11 PM
natekupp edited the test plan for this revision. (Show Details)
natekupp added inline comments.Thu, Sep 5, 5:41 PM
.buildkite/step_builder.py
18

will update this in a follow-up diff after I deploy v5

natekupp updated this revision to Diff 4281.Thu, Sep 5, 5:43 PM
natekupp edited the test plan for this revision. (Show Details)

spark 2.4.4

max accepted this revision.Thu, Sep 5, 5:52 PM
max added a subscriber: max.
max added inline comments.
.buildkite/images/Makefile
2

if we can use the most recent minor versions (on -stretch) we should

.buildkite/images/docker/snapshot-update.sh
25

do we think this stuff is only safe to do in a clean clone of the repo?

This revision is now accepted and ready to land.Thu, Sep 5, 5:52 PM
natekupp added inline comments.Thu, Sep 5, 8:08 PM
.buildkite/images/docker/snapshot-update.sh
25

yeah, otherwise it'll pick up other random stuff you've installed during the pip freeze - although I could leave around the snapshot-reqs-* venvs, but seems like the sort of thing to be "better safe than sorry" about since we do it so rarely