Tue, May 26
- move task output assignment into seperate function
Mon, May 25
- fix docker_build.sh to pass CI
I have this working with multi-step DAGs using execute plan while retrieving and storing values in the intermediate store.
I will test this with an S3 Intermediate Store. We should also figure out an architecture for allowing config data for StepInputs to be overridden by Flyte.
- removes ECR specific config
- defaultdict for compute_dict
- running on AWS KOPS cluster
- update output_name in compiler
- remove test file
Sat, May 23
remove dagster_aws from setup.py
make flyte tox runs more verbose
Fri, May 22
remove unnecessary intermediates_manger creation - save
ads dagit to setup.py - save
I am still testing this on multi step pipelines on Flyte infra. Expect a few more updates.
use execute plan and intermediates manager - save
Wed, May 20
- This should be able to compile arbitrary DAGs into a corresponding representation in Flyte.
- This also works with solids that have StepInputs with source_types from Multiple Outputs.
- There are a few type system quirks addressed in comments.
- I have addressed the Any type feedback. However see my note on this below.
- I am planning to have the execution_plan and Intermediate store work done Wednesday night.
removes raising an error on caching without flyte types
addresses type system feedback, allows definition of arbitrary DAG, as well as inputs from multiple outputs
Fri, May 15
Summarize our convo:
- Need to call execute plan, not the execution function
- Should support any type in dagster, and bypass the flyte type system
- Always persist everything to our intermediate store, for simplicity.
Critical to be able to demonstrate dagit side-by-side with flyte in a prod context.
addresses feedback, uses key intead of display_name for type hashing
check key in dict, not in list
Thu, May 14
make environment_dict required for compile_pipeline_to_flyte
executes successfully, sets outputs correctly, addresses feedback
remove hashing of ScalarTypes, use display_name attribute instead.
Wed, May 13
eq2 to other
Thu, May 7
Tue, May 5
adds version check to CI
Fri, May 1
build_docker -> docker_build
Only build flyte example once, for python 3.6
remove dagster-flyte untill it is in PyPI
add dagster-flyte to build kite install test. parameterize python version in flyte example container
moves create_plan_iterator call into get_sdk_node. move the set up and construction of SdkWorkflow into __call__
Thu, Apr 30
Apr 30 2020
install dagster-flyte in make install_dev_python_modules
add doc string
- install flytekit
- exclude tests
creates execution_plan and iterator when needed
Apr 29 2020
adds instance checks to flyte compiler. Removes requirements.txt, and utilizes setup.py
Apr 28 2020
newline between commans
tabs instead of spaces in makefile :(
add a CI test, that builds an example flyte container
calls super in a python2.7 compatible way
removes rogue typehint
flytekit version 0.6.2
use flytekit version 0.7.0
add flytekit to requirements
uses create library; adds CI
Apr 25 2020
The linter is currently failing because of an import error in example code in examples.py. Is there a good way to make it ignore that file?
Adds an example, rebases on master, cleans up imports, and removes type hints. tiny refactor
adds example, uses PipelineRun constructor.
Apr 24 2020
Apr 23 2020
removes exec, inherit from SdkWorkflow, add task attrs to current module
Apr 20 2020
remove use of exec
Use execute_plan_iterator, verified that flyte can execute
Apr 15 2020
Remove incorrect typehints
Wroking SdkWorkflow object and passing unit test