Page MenuHomePhabricator

jordanbramble (Jordan Bramble)
User

Projects

User does not belong to any projects.

User Details

User Since
Mar 30 2020, 4:34 PM (13 w, 4 d)

Recent Activity

Mon, Jun 15

jordanbramble committed R1:35467fd53590: adds ssh_resource changes to CHANGES.md (authored by jordanbramble).
adds ssh_resource changes to CHANGES.md
Mon, Jun 15, 8:35 PM
jordanbramble closed D3498: adds ssh_resource changes to CHANGES.md.
Mon, Jun 15, 8:35 PM
jordanbramble requested review of D3498: adds ssh_resource changes to CHANGES.md.
Mon, Jun 15, 8:32 PM
jordanbramble committed R1:f09dd60b49a5: removes sftp_solid in favor of adding functions to SSHResource (authored by jordanbramble).
removes sftp_solid in favor of adding functions to SSHResource
Mon, Jun 15, 7:21 PM
jordanbramble closed D3461: removes sftp_solid in favor of adding functions to SSHResource.
Mon, Jun 15, 7:21 PM
jordanbramble updated the diff for D3461: removes sftp_solid in favor of adding functions to SSHResource.

adds SSHResource to top level library, adds to docs

Mon, Jun 15, 6:22 PM

Fri, Jun 12

jordanbramble added reviewers for D3461: removes sftp_solid in favor of adding functions to SSHResource: sandyryza, max.
Fri, Jun 12, 11:13 PM
jordanbramble added a comment to D3461: removes sftp_solid in favor of adding functions to SSHResource.

@nate Fixed! This is good to go.

Fri, Jun 12, 9:22 PM
jordanbramble updated the diff for D3461: removes sftp_solid in favor of adding functions to SSHResource.

remove unused import

Fri, Jun 12, 9:06 PM
jordanbramble updated the diff for D3461: removes sftp_solid in favor of adding functions to SSHResource.

manually close SSHClient connection

Fri, Jun 12, 9:05 PM

Thu, Jun 11

jordanbramble committed R1:6bf17ba0d249: removes dagster-azure from docs_requirements.txt (authored by jordanbramble).
removes dagster-azure from docs_requirements.txt
Thu, Jun 11, 10:22 PM
jordanbramble closed D3463: removes dagster-azure from docs_requirements.txt.
Thu, Jun 11, 10:22 PM
jordanbramble requested review of D3461: removes sftp_solid in favor of adding functions to SSHResource.
Thu, Jun 11, 10:15 PM
jordanbramble requested review of D3463: removes dagster-azure from docs_requirements.txt.
Thu, Jun 11, 9:18 PM
jordanbramble committed R1:efec1519d6f7: remove BigQueryClient, extend gcp docs (authored by jordanbramble).
remove BigQueryClient, extend gcp docs
Thu, Jun 11, 6:35 PM
jordanbramble closed D3424: remove BigQueryClient, extend gcp docs.
Thu, Jun 11, 6:35 PM
jordanbramble committed R1:b5df7b0eee89: adds __all__ to library __init__.py (authored by jordanbramble).
adds __all__ to library __init__.py
Thu, Jun 11, 5:15 AM
jordanbramble closed D3436: adds __all__ to library __init__.py.
Thu, Jun 11, 5:15 AM
jordanbramble updated the diff for D3436: adds __all__ to library __init__.py.

rebase

Thu, Jun 11, 4:28 AM
jordanbramble requested review of D3436: adds __all__ to library __init__.py.
Thu, Jun 11, 3:53 AM
jordanbramble added a reviewer for D3424: remove BigQueryClient, extend gcp docs: schrockn.
Thu, Jun 11, 3:30 AM
jordanbramble updated the diff for D3424: remove BigQueryClient, extend gcp docs.

address feedback, remove changelog addition in favor of Max's PR

Thu, Jun 11, 2:58 AM
jordanbramble added inline comments to D3433: Changelog for 0.8.0.
Thu, Jun 11, 2:46 AM
jordanbramble requested review of D3424: remove BigQueryClient, extend gcp docs.
Thu, Jun 11, 12:39 AM

Wed, Jun 10

jordanbramble committed R1:fa276cc61607: Delete dagster_dbt (authored by jordanbramble).
Delete dagster_dbt
Wed, Jun 10, 9:04 PM
jordanbramble closed D3401: Delete dagster_dbt.
Wed, Jun 10, 9:04 PM
jordanbramble requested review of D3401: Delete dagster_dbt.
Wed, Jun 10, 8:49 PM
jordanbramble accepted D3398: (1/2) Refresh tutorial.

Looks good to me once CI passes!

Wed, Jun 10, 5:59 PM
jordanbramble added inline comments to D3398: (1/2) Refresh tutorial.
Wed, Jun 10, 4:56 PM
jordanbramble committed R1:061992ab5228: adds missing objects to autodoc files (authored by jordanbramble).
adds missing objects to autodoc files
Wed, Jun 10, 4:10 PM
jordanbramble closed D3394: adds missing objects to autodoc files.
Wed, Jun 10, 4:10 PM
jordanbramble requested review of D3394: adds missing objects to autodoc files.
Wed, Jun 10, 8:18 AM

Tue, Jun 9

jordanbramble added a comment to D3355: dagster-bash -> dagster-shell.

I will close this: https://dagster.phacility.com/D3361 In favor of your PR

Tue, Jun 9, 7:11 PM
jordanbramble committed R1:499e11006837: restores S3FakeSession, in favor of a more involved change control decision (authored by jordanbramble).
restores S3FakeSession, in favor of a more involved change control decision
Tue, Jun 9, 7:32 AM
jordanbramble closed D3360: restores S3FakeSession, in favor of a more involved change control decision.
Tue, Jun 9, 7:32 AM
jordanbramble requested review of D3360: restores S3FakeSession, in favor of a more involved change control decision.
Tue, Jun 9, 5:55 AM
jordanbramble committed R1:d8cfffadac79: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger (authored by jordanbramble).
removes create_s3_session, renames create_s3_fake_resource, renames S3Logger
Tue, Jun 9, 1:14 AM
jordanbramble closed D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.
Tue, Jun 9, 1:14 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

repin version, if works, won't require environment markers

Tue, Jun 9, 12:48 AM

Mon, Jun 8

jordanbramble added a comment to D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

I think I have some evidence that moving the list_objects operation up is fixing this. I would have to dive deeper, but I think result_for_solid() is interacting with boto and perhaps redeclaring the client. I would be comfortable merging this and then if we continue to see this flake arise, we can treat it as a p1, or disable and treat as a p2.

Mon, Jun 8, 8:21 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

try re-arranging order in case boto client is being overwritten

Mon, Jun 8, 7:55 AM
jordanbramble updated the test plan for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.
Mon, Jun 8, 7:40 AM
jordanbramble added a comment to D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

@nate I am currently battling test flakes with Moto where occasionally it will actually make network calls to AWS and then get access denied errors on python 3.5 Funnily enough, when searching GitHub issues for Moto, I saw you mentioned here: https://github.com/spulec/moto/issues/2118
and you referenced this diff: https://dagster.phacility.com/D350

Mon, Jun 8, 7:40 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

testing if unpinning moto version fixes test flake on python 3.5

Mon, Jun 8, 7:20 AM

Sat, Jun 6

jordanbramble added a comment to D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

Tests are passing now, but test_depends_on_s3_resource_intermediates seems to flake with an access denied on list objects, pretty often. I am not sure why we would get a permissions error when using mock_s3.

Sat, Jun 6, 6:32 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

fix snapshot tests

Sat, Jun 6, 5:52 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

fix snapshot test

Sat, Jun 6, 5:34 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

fixes test errors

Sat, Jun 6, 4:58 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

fixing tests to work with moto mock_S3

Sat, Jun 6, 4:44 AM

Fri, Jun 5

jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

fix airline demo tests to use keyword arguments when creating s3 buckets

Fri, Jun 5, 11:36 PM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

add moto to setup.py

Fri, Jun 5, 11:20 PM
jordanbramble added inline comments to D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.
Fri, Jun 5, 11:13 PM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

removes fake s3 sessions and replaces with @mock_s3 decorator, updates docs to reflect

Fri, Jun 5, 11:10 PM
jordanbramble added a reviewer for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger: nate.
Fri, Jun 5, 9:16 PM
jordanbramble added inline comments to D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.
Fri, Jun 5, 9:15 PM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

replaces fake_s3_resource with moto

Fri, Jun 5, 9:12 PM
jordanbramble added inline comments to D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.
Fri, Jun 5, 7:41 PM
jordanbramble requested review of D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.
Fri, Jun 5, 8:44 AM

Thu, Jun 4

jordanbramble committed R1:00a5957a4c4f: liberally replace str/String -> StringSource and int/Int -> IntSource (authored by jordanbramble).
liberally replace str/String -> StringSource and int/Int -> IntSource
Thu, Jun 4, 8:54 PM
jordanbramble closed D3268: liberally replace str/String -> StringSource and int/Int -> IntSource.
Thu, Jun 4, 8:54 PM
jordanbramble accepted D3247: More library docs updates.
Thu, Jun 4, 5:19 AM
jordanbramble added inline comments to D3247: More library docs updates.
Thu, Jun 4, 5:19 AM
jordanbramble added a reviewer for D3268: liberally replace str/String -> StringSource and int/Int -> IntSource: nate.
Thu, Jun 4, 5:06 AM
jordanbramble requested review of D3268: liberally replace str/String -> StringSource and int/Int -> IntSource.
Thu, Jun 4, 4:53 AM

Jun 3 2020

jordanbramble committed R1:d0497c0aa262: use StringSource with Fields in resource decorator (authored by jordanbramble).
use StringSource with Fields in resource decorator
Jun 3 2020, 4:38 AM
jordanbramble closed D3244: use StringSource with Fields in resource decorator.
Jun 3 2020, 4:38 AM
jordanbramble added a reviewer for D3244: use StringSource with Fields in resource decorator: nate.
Jun 3 2020, 3:33 AM
jordanbramble requested review of D3244: use StringSource with Fields in resource decorator.
Jun 3 2020, 3:18 AM
jordanbramble accepted D3239: Add all library readmes and validate.
Jun 3 2020, 1:00 AM
jordanbramble added inline comments to D3239: Add all library readmes and validate.
Jun 3 2020, 12:40 AM
jordanbramble closed D3237: remove unpacking of values, explicity get dict values to create twilio client.
Jun 3 2020, 12:00 AM
jordanbramble committed R1:a85e145e50df: remove unpacking of values, explicity get dict values to create twilio client (authored by jordanbramble).
remove unpacking of values, explicity get dict values to create twilio client
Jun 3 2020, 12:00 AM

Jun 2 2020

jordanbramble added a comment to D3237: remove unpacking of values, explicity get dict values to create twilio client.

Dictionaries in python are not guaranteed to preserve insert order before 3.6. so here I am explictly getting the account_sid and auth_token in order instead of unpacking the values. We could double splat but we need to rename the resource config keys to match Twilio's Client init's param names.

Jun 2 2020, 11:56 PM
jordanbramble requested review of D3237: remove unpacking of values, explicity get dict values to create twilio client.
Jun 2 2020, 11:48 PM
jordanbramble committed R1:4f3caafd1c8e: removes unneccessary TwilioClient, corrects description, and utilizes… (authored by jordanbramble).
removes unneccessary TwilioClient, corrects description, and utilizes…
Jun 2 2020, 10:14 PM
jordanbramble closed D3218: removes unneccessary TwilioClient, corrects description, and utilizes StringSource.
Jun 2 2020, 10:14 PM
jordanbramble requested review of D3218: removes unneccessary TwilioClient, corrects description, and utilizes StringSource.
Jun 2 2020, 5:18 AM

May 26 2020

jordanbramble committed R1:79d37d18a9ea: Implements Flyte Type System mapping (authored by jordanbramble).
Implements Flyte Type System mapping
May 26 2020, 7:18 PM
jordanbramble closed D2901: Implements Flyte Type System mapping.
May 26 2020, 7:18 PM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
May 26 2020, 6:30 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • move task output assignment into seperate function
May 26 2020, 6:27 AM

May 25 2020

jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • fix docker_build.sh to pass CI
May 25 2020, 6:34 AM
jordanbramble added a comment to D2901: Implements Flyte Type System mapping.

I have this working with multi-step DAGs using execute plan while retrieving and storing values in the intermediate store.
I will test this with an S3 Intermediate Store. We should also figure out an architecture for allowing config data for StepInputs to be overridden by Flyte.

May 25 2020, 6:23 AM
jordanbramble updated the summary of D2901: Implements Flyte Type System mapping.
May 25 2020, 6:16 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • removes ECR specific config
May 25 2020, 6:10 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • defaultdict for compute_dict
May 25 2020, 6:08 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • running on AWS KOPS cluster
  • update output_name in compiler
  • remove test file
May 25 2020, 5:47 AM

May 23 2020

jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

remove dagster_aws from setup.py

May 23 2020, 12:56 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

up

May 23 2020, 12:26 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

make flyte tox runs more verbose

May 23 2020, 12:12 AM

May 22 2020

jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

remove unnecessary intermediates_manger creation - save

May 22 2020, 10:09 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

ads dagit to setup.py - save

May 22 2020, 9:56 PM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
May 22 2020, 9:51 PM
jordanbramble added a comment to D2901: Implements Flyte Type System mapping.

I am still testing this on multi step pipelines on Flyte infra. Expect a few more updates.

May 22 2020, 8:38 PM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
May 22 2020, 8:37 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

up

May 22 2020, 8:32 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

use execute plan and intermediates manager - save

May 22 2020, 8:29 PM

May 20 2020

jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
May 20 2020, 9:35 AM
jordanbramble added a comment to D2901: Implements Flyte Type System mapping.
  1. This should be able to compile arbitrary DAGs into a corresponding representation in Flyte.
  2. This also works with solids that have StepInputs with source_types from Multiple Outputs.
  3. There are a few type system quirks addressed in comments.
  4. I have addressed the Any type feedback. However see my note on this below.
  5. I am planning to have the execution_plan and Intermediate store work done Wednesday night.
May 20 2020, 9:34 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

removes raising an error on caching without flyte types

May 20 2020, 8:50 AM