This prototype hooks in image handles to the dagit CLI as well as introduces a dagster container snapshot CLI. You can now build your user code as a container and as long as your Dockerfile provides a ENTRYPOINT ["/usr/local/bin/dagster"] entrypoint. You are good to go. To watch this work, take the following Dockerfile which represents the airline demo.
FROM dagster/dagster-py37:latest RUN apt-get update \ && mkdir -p /opt/dagster/ \ && mkdir -p /opt/dagster/dagster_home ADD ./dagster /opt/dagster WORKDIR /opt/dagster RUN pip install --upgrade pip \ && make install_dev_python_modules WORKDIR /opt/dagster/examples/dagster_examples/airline_demo ENTRYPOINT ["/usr/local/bin/dagster"]
Then build an image by running docker build -t airline_user_code -f <path to dockerfile> .
Once your image is built. You can from anywhere run: dagit -p 3333 --image airline_user_code and everything should work!
- This really only works with Airlines, I didn't wan't to waste too much time on this because Nick and Alex are building their own snapshots.
- I am noticing that the whole ExecutionTargetHandle/ExecutionTargetData/Loader interface breaks down a bit once containers get thrown in. I was just cutting corners to get stuff working but I would love to learn more about what their abstraction boundaries are (why seperate handle v data)?
- Execution isn't "isolated" since we are serializing code, I haven't gotten around to the ContainerExecutionManager which I will build in tomorrow but it shouldn't be hard.
But would love comments and figured this would be useful for our architecture meeting!