Flytekit Proof of Concept

Authored by jordanbramble on Thu, Apr 30, 2:27 AM.


Flytekit Proof of Concept

For an architecture diagram that explains the logic behind the DagsterFlyteSdkWorflow class please review.

This diff implements a first pass of the class shaded in blue that inherits from a flytekit workflow. I have done this in an object oriented fashion, but provided a function that encapsulates everything. Please let me know if people would prefer something less object oriented here.

Currently, this iteration bypasses Flyte's type system as well as their Input + Output promises. To that end, this will only work when all of the input and output config is provided adequately via Dagster. However, I believe so long as we do this, then Flyte can run arbitrary configurations of our DAGs.

In a future diff we can map the input/output promises as well as types. The flyte team is willing to offer some guidance on working with their type system.

Test Plan:
pytest python_modules/libraries/dagster-flyte/dagster_flyte_tests/test_dagster_flyte.py

In order to test in kubernetes do the following:

  1. Follow the Installation guide here.
  2. minikube service --alsologtostderr -v=3 contour -n heptio-contour
  3. minikube ssh
  4. You will need to install git and potentially a text editor on your minikube instance.
  5. Copy up the examples directory contained in the docker-flyte library.
  6. curl -X POST localhost:30081/api/v1/projects -d '{"project": {"id": "dagstertest", "name": "dagstertest"} }'
  7. make docker_build
  8. run docker image ls and grab the image id of the image you just built.
  9. docker run --network host -e FLYTE_PLATFORM_URL='' {{ your docker image }} pyflyte -p dagstertest -d development -c sandbox.config register workflows
  10. Navigate to , find your workflow and launch it!

Reviewers: nate, schrockn

Reviewed By: schrockn

Differential Revision: https://dagster.phacility.com/D2544


jordanbrambleFri, May 1, 5:18 PM
Differential Revision
D2544: Flytekit Proof of Concept
R1:0bbf495cc592: update schedule and partition callsites