Page MenuHomePhabricator

jordanbramble (Jordan Bramble)
Disabled

Projects

User does not belong to any projects.

User Details

User Since
Mar 30 2020, 4:34 PM (24 w, 6 d)
Roles
Disabled

Recent Activity

Jun 15 2020

jordanbramble committed R1:35467fd53590: adds ssh_resource changes to CHANGES.md (authored by jordanbramble).
adds ssh_resource changes to CHANGES.md
Jun 15 2020, 8:35 PM
jordanbramble closed D3498: adds ssh_resource changes to CHANGES.md.
Jun 15 2020, 8:35 PM
jordanbramble requested review of D3498: adds ssh_resource changes to CHANGES.md.
Jun 15 2020, 8:32 PM
jordanbramble committed R1:f09dd60b49a5: removes sftp_solid in favor of adding functions to SSHResource (authored by jordanbramble).
removes sftp_solid in favor of adding functions to SSHResource
Jun 15 2020, 7:21 PM
jordanbramble closed D3461: removes sftp_solid in favor of adding functions to SSHResource.
Jun 15 2020, 7:21 PM
jordanbramble updated the diff for D3461: removes sftp_solid in favor of adding functions to SSHResource.

adds SSHResource to top level library, adds to docs

Jun 15 2020, 6:22 PM

Jun 12 2020

jordanbramble added reviewers for D3461: removes sftp_solid in favor of adding functions to SSHResource: sandyryza, max.
Jun 12 2020, 11:13 PM
jordanbramble added a comment to D3461: removes sftp_solid in favor of adding functions to SSHResource.

@nate Fixed! This is good to go.

Jun 12 2020, 9:22 PM
jordanbramble updated the diff for D3461: removes sftp_solid in favor of adding functions to SSHResource.

remove unused import

Jun 12 2020, 9:06 PM
jordanbramble updated the diff for D3461: removes sftp_solid in favor of adding functions to SSHResource.

manually close SSHClient connection

Jun 12 2020, 9:05 PM

Jun 11 2020

jordanbramble committed R1:6bf17ba0d249: removes dagster-azure from docs_requirements.txt (authored by jordanbramble).
removes dagster-azure from docs_requirements.txt
Jun 11 2020, 10:22 PM
jordanbramble closed D3463: removes dagster-azure from docs_requirements.txt.
Jun 11 2020, 10:22 PM
jordanbramble requested review of D3461: removes sftp_solid in favor of adding functions to SSHResource.
Jun 11 2020, 10:15 PM
jordanbramble requested review of D3463: removes dagster-azure from docs_requirements.txt.
Jun 11 2020, 9:18 PM
jordanbramble committed R1:efec1519d6f7: remove BigQueryClient, extend gcp docs (authored by jordanbramble).
remove BigQueryClient, extend gcp docs
Jun 11 2020, 6:35 PM
jordanbramble closed D3424: remove BigQueryClient, extend gcp docs.
Jun 11 2020, 6:35 PM
jordanbramble committed R1:b5df7b0eee89: adds __all__ to library __init__.py (authored by jordanbramble).
adds __all__ to library __init__.py
Jun 11 2020, 5:15 AM
jordanbramble closed D3436: adds __all__ to library __init__.py.
Jun 11 2020, 5:15 AM
jordanbramble updated the diff for D3436: adds __all__ to library __init__.py.

rebase

Jun 11 2020, 4:28 AM
jordanbramble requested review of D3436: adds __all__ to library __init__.py.
Jun 11 2020, 3:53 AM
jordanbramble added a reviewer for D3424: remove BigQueryClient, extend gcp docs: schrockn.
Jun 11 2020, 3:30 AM
jordanbramble updated the diff for D3424: remove BigQueryClient, extend gcp docs.

address feedback, remove changelog addition in favor of Max's PR

Jun 11 2020, 2:58 AM
jordanbramble added inline comments to D3433: Changelog for 0.8.0.
Jun 11 2020, 2:46 AM
jordanbramble requested review of D3424: remove BigQueryClient, extend gcp docs.
Jun 11 2020, 12:39 AM

Jun 10 2020

jordanbramble committed R1:fa276cc61607: Delete dagster_dbt (authored by jordanbramble).
Delete dagster_dbt
Jun 10 2020, 9:04 PM
jordanbramble closed D3401: Delete dagster_dbt.
Jun 10 2020, 9:04 PM
jordanbramble requested review of D3401: Delete dagster_dbt.
Jun 10 2020, 8:49 PM
jordanbramble accepted D3398: (1/2) Refresh tutorial.

Looks good to me once CI passes!

Jun 10 2020, 5:59 PM
jordanbramble added inline comments to D3398: (1/2) Refresh tutorial.
Jun 10 2020, 4:56 PM
jordanbramble committed R1:061992ab5228: adds missing objects to autodoc files (authored by jordanbramble).
adds missing objects to autodoc files
Jun 10 2020, 4:10 PM
jordanbramble closed D3394: adds missing objects to autodoc files.
Jun 10 2020, 4:10 PM
jordanbramble requested review of D3394: adds missing objects to autodoc files.
Jun 10 2020, 8:18 AM

Jun 9 2020

jordanbramble added a comment to D3355: dagster-bash -> dagster-shell.

I will close this: https://dagster.phacility.com/D3361 In favor of your PR

Jun 9 2020, 7:11 PM
jordanbramble committed R1:499e11006837: restores S3FakeSession, in favor of a more involved change control decision (authored by jordanbramble).
restores S3FakeSession, in favor of a more involved change control decision
Jun 9 2020, 7:32 AM
jordanbramble closed D3360: restores S3FakeSession, in favor of a more involved change control decision.
Jun 9 2020, 7:32 AM
jordanbramble requested review of D3360: restores S3FakeSession, in favor of a more involved change control decision.
Jun 9 2020, 5:55 AM
jordanbramble committed R1:d8cfffadac79: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger (authored by jordanbramble).
removes create_s3_session, renames create_s3_fake_resource, renames S3Logger
Jun 9 2020, 1:14 AM
jordanbramble closed D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.
Jun 9 2020, 1:14 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

repin version, if works, won't require environment markers

Jun 9 2020, 12:48 AM

Jun 8 2020

jordanbramble added a comment to D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

I think I have some evidence that moving the list_objects operation up is fixing this. I would have to dive deeper, but I think result_for_solid() is interacting with boto and perhaps redeclaring the client. I would be comfortable merging this and then if we continue to see this flake arise, we can treat it as a p1, or disable and treat as a p2.

Jun 8 2020, 8:21 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

try re-arranging order in case boto client is being overwritten

Jun 8 2020, 7:55 AM
jordanbramble updated the test plan for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.
Jun 8 2020, 7:40 AM
jordanbramble added a comment to D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

@nate I am currently battling test flakes with Moto where occasionally it will actually make network calls to AWS and then get access denied errors on python 3.5 Funnily enough, when searching GitHub issues for Moto, I saw you mentioned here: https://github.com/spulec/moto/issues/2118
and you referenced this diff: https://dagster.phacility.com/D350

Jun 8 2020, 7:40 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

testing if unpinning moto version fixes test flake on python 3.5

Jun 8 2020, 7:20 AM

Jun 6 2020

jordanbramble added a comment to D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

Tests are passing now, but test_depends_on_s3_resource_intermediates seems to flake with an access denied on list objects, pretty often. I am not sure why we would get a permissions error when using mock_s3.

Jun 6 2020, 6:32 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

fix snapshot tests

Jun 6 2020, 5:52 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

fix snapshot test

Jun 6 2020, 5:34 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

fixes test errors

Jun 6 2020, 4:58 AM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

fixing tests to work with moto mock_S3

Jun 6 2020, 4:44 AM

Jun 5 2020

jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

fix airline demo tests to use keyword arguments when creating s3 buckets

Jun 5 2020, 11:36 PM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

add moto to setup.py

Jun 5 2020, 11:20 PM
jordanbramble added inline comments to D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.
Jun 5 2020, 11:13 PM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

removes fake s3 sessions and replaces with @mock_s3 decorator, updates docs to reflect

Jun 5 2020, 11:10 PM
jordanbramble added a reviewer for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger: nate.
Jun 5 2020, 9:16 PM
jordanbramble added inline comments to D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.
Jun 5 2020, 9:15 PM
jordanbramble updated the diff for D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.

replaces fake_s3_resource with moto

Jun 5 2020, 9:12 PM
jordanbramble added inline comments to D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.
Jun 5 2020, 7:41 PM
jordanbramble requested review of D3294: removes create_s3_session, renames create_s3_fake_resource, renames S3Logger.
Jun 5 2020, 8:44 AM

Jun 4 2020

jordanbramble committed R1:00a5957a4c4f: liberally replace str/String -> StringSource and int/Int -> IntSource (authored by jordanbramble).
liberally replace str/String -> StringSource and int/Int -> IntSource
Jun 4 2020, 8:54 PM
jordanbramble closed D3268: liberally replace str/String -> StringSource and int/Int -> IntSource.
Jun 4 2020, 8:54 PM
jordanbramble accepted D3247: More library docs updates.
Jun 4 2020, 5:19 AM
jordanbramble added inline comments to D3247: More library docs updates.
Jun 4 2020, 5:19 AM
jordanbramble added a reviewer for D3268: liberally replace str/String -> StringSource and int/Int -> IntSource: nate.
Jun 4 2020, 5:06 AM
jordanbramble requested review of D3268: liberally replace str/String -> StringSource and int/Int -> IntSource.
Jun 4 2020, 4:53 AM

Jun 3 2020

jordanbramble committed R1:d0497c0aa262: use StringSource with Fields in resource decorator (authored by jordanbramble).
use StringSource with Fields in resource decorator
Jun 3 2020, 4:38 AM
jordanbramble closed D3244: use StringSource with Fields in resource decorator.
Jun 3 2020, 4:38 AM
jordanbramble added a reviewer for D3244: use StringSource with Fields in resource decorator: nate.
Jun 3 2020, 3:33 AM
jordanbramble requested review of D3244: use StringSource with Fields in resource decorator.
Jun 3 2020, 3:18 AM
jordanbramble accepted D3239: Add all library readmes and validate.
Jun 3 2020, 1:00 AM
jordanbramble added inline comments to D3239: Add all library readmes and validate.
Jun 3 2020, 12:40 AM
jordanbramble closed D3237: remove unpacking of values, explicity get dict values to create twilio client.
Jun 3 2020, 12:00 AM
jordanbramble committed R1:a85e145e50df: remove unpacking of values, explicity get dict values to create twilio client (authored by jordanbramble).
remove unpacking of values, explicity get dict values to create twilio client
Jun 3 2020, 12:00 AM

Jun 2 2020

jordanbramble added a comment to D3237: remove unpacking of values, explicity get dict values to create twilio client.

Dictionaries in python are not guaranteed to preserve insert order before 3.6. so here I am explictly getting the account_sid and auth_token in order instead of unpacking the values. We could double splat but we need to rename the resource config keys to match Twilio's Client init's param names.

Jun 2 2020, 11:56 PM
jordanbramble requested review of D3237: remove unpacking of values, explicity get dict values to create twilio client.
Jun 2 2020, 11:48 PM
jordanbramble committed R1:4f3caafd1c8e: removes unneccessary TwilioClient, corrects description, and utilizes… (authored by jordanbramble).
removes unneccessary TwilioClient, corrects description, and utilizes…
Jun 2 2020, 10:14 PM
jordanbramble closed D3218: removes unneccessary TwilioClient, corrects description, and utilizes StringSource.
Jun 2 2020, 10:14 PM
jordanbramble requested review of D3218: removes unneccessary TwilioClient, corrects description, and utilizes StringSource.
Jun 2 2020, 5:18 AM

May 26 2020

jordanbramble committed R1:79d37d18a9ea: Implements Flyte Type System mapping (authored by jordanbramble).
Implements Flyte Type System mapping
May 26 2020, 7:18 PM
jordanbramble closed D2901: Implements Flyte Type System mapping.
May 26 2020, 7:18 PM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
May 26 2020, 6:30 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • move task output assignment into seperate function
May 26 2020, 6:27 AM

May 25 2020

jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • fix docker_build.sh to pass CI
May 25 2020, 6:34 AM
jordanbramble added a comment to D2901: Implements Flyte Type System mapping.

I have this working with multi-step DAGs using execute plan while retrieving and storing values in the intermediate store.
I will test this with an S3 Intermediate Store. We should also figure out an architecture for allowing config data for StepInputs to be overridden by Flyte.

May 25 2020, 6:23 AM
jordanbramble updated the summary of D2901: Implements Flyte Type System mapping.
May 25 2020, 6:16 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • removes ECR specific config
May 25 2020, 6:10 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • defaultdict for compute_dict
May 25 2020, 6:08 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.
  • running on AWS KOPS cluster
  • update output_name in compiler
  • remove test file
May 25 2020, 5:47 AM

May 23 2020

jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

remove dagster_aws from setup.py

May 23 2020, 12:56 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

up

May 23 2020, 12:26 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

make flyte tox runs more verbose

May 23 2020, 12:12 AM

May 22 2020

jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

remove unnecessary intermediates_manger creation - save

May 22 2020, 10:09 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

ads dagit to setup.py - save

May 22 2020, 9:56 PM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
May 22 2020, 9:51 PM
jordanbramble added a comment to D2901: Implements Flyte Type System mapping.

I am still testing this on multi step pipelines on Flyte infra. Expect a few more updates.

May 22 2020, 8:38 PM
jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
May 22 2020, 8:37 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

up

May 22 2020, 8:32 PM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

use execute plan and intermediates manager - save

May 22 2020, 8:29 PM

May 20 2020

jordanbramble added inline comments to D2901: Implements Flyte Type System mapping.
May 20 2020, 9:35 AM
jordanbramble added a comment to D2901: Implements Flyte Type System mapping.
  1. This should be able to compile arbitrary DAGs into a corresponding representation in Flyte.
  2. This also works with solids that have StepInputs with source_types from Multiple Outputs.
  3. There are a few type system quirks addressed in comments.
  4. I have addressed the Any type feedback. However see my note on this below.
  5. I am planning to have the execution_plan and Intermediate store work done Wednesday night.
May 20 2020, 9:34 AM
jordanbramble updated the diff for D2901: Implements Flyte Type System mapping.

removes raising an error on caching without flyte types

May 20 2020, 8:50 AM