Page MenuHomePhabricator

jordanbramble (Jordan Bramble)
User

Projects

User does not belong to any projects.

User Details

User Since
Mar 30 2020, 4:34 PM (8 w, 5 d)

Recent Activity

Tue, May 26

jordanbramble committed R1:79d37d18a9ea: Implements Flyte Type System mapping (authored by jordanbramble).
Implements Flyte Type System mapping
Tue, May 26, 7:18 PM
jordanbramble closed D2901: Implements Flyte Type System mapping.
Tue, May 26, 7:18 PM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
Tue, May 26, 6:30 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • move task output assignment into seperate function
Tue, May 26, 6:27 AM

Mon, May 25

jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • fix docker_build.sh to pass CI
Mon, May 25, 6:34 AM
jordanbramble added a comment to D2901: Implements Flyte Type System mapping.

I have this working with multi-step DAGs using execute plan while retrieving and storing values in the intermediate store.
I will test this with an S3 Intermediate Store. We should also figure out an architecture for allowing config data for StepInputs to be overridden by Flyte.

Mon, May 25, 6:23 AM
jordanbramble updated the summary of D2901: Implements Flyte Type System mapping.
Mon, May 25, 6:16 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • removes ECR specific config
Mon, May 25, 6:10 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • defaultdict for compute_dict
Mon, May 25, 6:08 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • running on AWS KOPS cluster
  • update output_name in compiler
  • remove test file
Mon, May 25, 5:47 AM

Sat, May 23

jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

remove dagster_aws from setup.py

Sat, May 23, 12:56 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

up

Sat, May 23, 12:26 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

make flyte tox runs more verbose

Sat, May 23, 12:12 AM

Fri, May 22

jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

remove unnecessary intermediates_manger creation - save

Fri, May 22, 10:09 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

ads dagit to setup.py - save

Fri, May 22, 9:56 PM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
Fri, May 22, 9:51 PM
jordanbramble added a comment to D2901: Implements Flyte Type System mapping.

I am still testing this on multi step pipelines on Flyte infra. Expect a few more updates.

Fri, May 22, 8:38 PM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
Fri, May 22, 8:37 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

up

Fri, May 22, 8:32 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

use execute plan and intermediates manager - save

Fri, May 22, 8:29 PM

Wed, May 20

jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
Wed, May 20, 9:35 AM
jordanbramble added a comment to D2901: Implements Flyte Type System mapping.
  1. This should be able to compile arbitrary DAGs into a corresponding representation in Flyte.
  2. This also works with solids that have StepInputs with source_types from Multiple Outputs.
  3. There are a few type system quirks addressed in comments.
  4. I have addressed the Any type feedback. However see my note on this below.
  5. I am planning to have the execution_plan and Intermediate store work done Wednesday night.
Wed, May 20, 9:34 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

removes raising an error on caching without flyte types

Wed, May 20, 8:50 AM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
Wed, May 20, 8:49 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

addresses type system feedback, allows definition of arbitrary DAG, as well as inputs from multiple outputs

Wed, May 20, 8:00 AM
jordanbramble added inline comments to D2996: Rename get_or_create_run() => register_managed_run() and other clean up.
Wed, May 20, 12:54 AM

Fri, May 15

jordanbramble added a comment to D2901: Implements Flyte Type System mapping.

Summarize our convo:

  1. Need to call execute plan, not the execution function
  2. Should support any type in dagster, and bypass the flyte type system
  3. Always persist everything to our intermediate store, for simplicity.

Critical to be able to demonstrate dagit side-by-side with flyte in a prod context.

Fri, May 15, 11:59 PM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
Fri, May 15, 7:27 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

addresses feedback, uses key intead of display_name for type hashing

Fri, May 15, 7:21 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

add README

Fri, May 15, 5:56 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

check key in dict, not in list

Fri, May 15, 3:06 AM

Thu, May 14

jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
Thu, May 14, 11:01 PM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
Thu, May 14, 10:41 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

make environment_dict required for compile_pipeline_to_flyte

Thu, May 14, 8:09 AM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
Thu, May 14, 7:54 AM
jordanbramble updated the test plan for D2901: Implements Flyte Type System mapping.
Thu, May 14, 7:34 AM
jordanbramble updated the summary of D2901: Implements Flyte Type System mapping.
Thu, May 14, 7:34 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

executes successfully, sets outputs correctly, addresses feedback

Thu, May 14, 7:29 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

remove hashing of ScalarTypes, use display_name attribute instead.

Thu, May 14, 4:33 AM

Wed, May 13

jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
Wed, May 13, 11:50 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

eq2 to other

Wed, May 13, 8:55 AM
jordanbramble retitled D2901: Implements Flyte Type System mapping from implements flyte type system support on inputs for dagster scalar types. outputs coming soon. to Implements Flyte Type System mapping.
Wed, May 13, 6:28 AM
jordanbramble requested review of D2901: Implements Flyte Type System mapping.
Wed, May 13, 6:13 AM

Thu, May 7

jordanbramble accepted D2813: Fix broken link to dagster types.

Looks good to me.

Thu, May 7, 5:50 AM
jordanbramble added inline comments to D2609: Dagster Celery K8s Job step executor.
Thu, May 7, 12:31 AM

Tue, May 5

jordanbramble committed R1:6ae6e72a7c71: bumps dagster-flyte version to be in sync with other libraries (authored by jordanbramble).
bumps dagster-flyte version to be in sync with other libraries
Tue, May 5, 6:54 PM
jordanbramble closed D2782: bumps dagster-flyte version to be in sync with other libraries.
Tue, May 5, 6:54 PM
jordanbramble added inline comments to D2782: bumps dagster-flyte version to be in sync with other libraries.
Tue, May 5, 5:59 PM
jordanbramble updated the test plan for D2782: bumps dagster-flyte version to be in sync with other libraries.
Tue, May 5, 5:57 PM
jordanbramble updated the diff for D2782: bumps dagster-flyte version to be in sync with other libraries.

adds version check to CI

Tue, May 5, 5:54 PM
jordanbramble requested review of D2782: bumps dagster-flyte version to be in sync with other libraries.
Tue, May 5, 5:27 PM

Fri, May 1

jordanbramble committed R1:95c32edc39bd: Flytekit Proof of Concept (authored by jordanbramble).
Flytekit Proof of Concept
Fri, May 1, 5:19 PM
jordanbramble closed D2544: Flytekit Proof of Concept.
Fri, May 1, 5:19 PM

May 1 2020

jordanbramble updated the diff for D2544: Flytekit Proof of Concept.
build_docker -> docker_build
May 1 2020, 3:18 AM
jordanbramble added inline comments to D2544: Flytekit Proof of Concept.
May 1 2020, 3:15 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

Only build flyte example once, for python 3.6

May 1 2020, 3:13 AM
jordanbramble added inline comments to D2544: Flytekit Proof of Concept.
May 1 2020, 2:57 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.
remove dagster-flyte untill it is in PyPI
May 1 2020, 2:52 AM
jordanbramble added inline comments to D2544: Flytekit Proof of Concept.
May 1 2020, 2:49 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.
add dagster-flyte to build kite install test. parameterize python version in flyte example container
May 1 2020, 2:37 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.
moves create_plan_iterator call into get_sdk_node. move the set up and construction of SdkWorkflow into __call__
May 1 2020, 1:12 AM

Apr 30 2020

jordanbramble added inline comments to D2544: Flytekit Proof of Concept.
Apr 30 2020, 5:45 PM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.
install dagster-flyte in make install_dev_python_modules
Apr 30 2020, 2:55 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.
add doc string
Apr 30 2020, 2:39 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.
  • install flytekit
Apr 30 2020, 2:28 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.
  • exclude tests
Apr 30 2020, 1:20 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

creates execution_plan and iterator when needed

Apr 30 2020, 12:23 AM

Apr 29 2020

jordanbramble updated the summary of D2544: Flytekit Proof of Concept.
Apr 29 2020, 4:01 AM
jordanbramble retitled D2544: Flytekit Proof of Concept from adds WIP bridge class for creating a flyte workflow that can be registered with a unit test that verifies class instantiation. to Flytekit Proof of Concept.
Apr 29 2020, 4:00 AM
jordanbramble added inline comments to D2544: Flytekit Proof of Concept.
Apr 29 2020, 4:00 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

adds instance checks to flyte compiler. Removes requirements.txt, and utilizes setup.py

Apr 29 2020, 3:59 AM
jordanbramble added inline comments to D2544: Flytekit Proof of Concept.
Apr 29 2020, 3:43 AM

Apr 28 2020

jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

newline between commans

Apr 28 2020, 8:45 PM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

tabs instead of spaces in makefile :(

Apr 28 2020, 8:32 PM
jordanbramble added reviewers for D2544: Flytekit Proof of Concept: nate, schrockn.
Apr 28 2020, 6:43 PM
jordanbramble added inline comments to D2544: Flytekit Proof of Concept.
Apr 28 2020, 6:41 PM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

add a CI test, that builds an example flyte container

Apr 28 2020, 6:32 PM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

calls super in a python2.7 compatible way

Apr 28 2020, 5:12 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

removes rogue typehint

Apr 28 2020, 5:02 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

flytekit version 0.6.2

Apr 28 2020, 4:55 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

use flytekit version 0.7.0

Apr 28 2020, 4:53 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

add flytekit to requirements

Apr 28 2020, 4:35 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

uses create library; adds CI

Apr 28 2020, 3:59 AM

Apr 25 2020

jordanbramble added inline comments to D2544: Flytekit Proof of Concept.
Apr 25 2020, 2:55 AM
jordanbramble updated the summary of D2544: Flytekit Proof of Concept.
Apr 25 2020, 2:42 AM
jordanbramble added a comment to D2544: Flytekit Proof of Concept.

The linter is currently failing because of an import error in example code in examples.py. Is there a good way to make it ignore that file?

Apr 25 2020, 2:37 AM
jordanbramble updated the test plan for D2544: Flytekit Proof of Concept.
Apr 25 2020, 2:29 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

Adds an example, rebases on master, cleans up imports, and removes type hints. tiny refactor

Apr 25 2020, 2:21 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

adds example, uses PipelineRun constructor.

Apr 25 2020, 1:58 AM

Apr 24 2020

jordanbramble added inline comments to D2544: Flytekit Proof of Concept.
Apr 24 2020, 12:48 AM

Apr 23 2020

jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

removes exec, inherit from SdkWorkflow, add task attrs to current module

Apr 23 2020, 1:22 AM

Apr 20 2020

jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

remove use of exec

Apr 20 2020, 6:37 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

Use execute_plan_iterator, verified that flyte can execute

Apr 20 2020, 5:32 AM

Apr 15 2020

jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

Remove incorrect typehints

Apr 15 2020, 5:51 AM
jordanbramble updated the diff for D2544: Flytekit Proof of Concept.

Wroking SdkWorkflow object and passing unit test

Apr 15 2020, 5:50 AM

Apr 13 2020

jordanbramble updated the summary of D2544: Flytekit Proof of Concept.
Apr 13 2020, 10:23 PM
jordanbramble added inline comments to D2544: Flytekit Proof of Concept.
Apr 13 2020, 10:13 PM
jordanbramble updated the summary of D2544: Flytekit Proof of Concept.
Apr 13 2020, 10:11 PM
jordanbramble created D2544: Flytekit Proof of Concept.
Apr 13 2020, 10:06 PM

Apr 3 2020

jordanbramble committed R1:c629d9aab8a7: Minor refactor of execute.py (authored by jordanbramble).
Minor refactor of execute.py
Apr 3 2020, 7:03 PM