- User Since
- Mar 30 2020, 4:34 PM (13 w, 4 d)
Mon, Jun 15
adds SSHResource to top level library, adds to docs
Fri, Jun 12
@nate Fixed! This is good to go.
remove unused import
manually close SSHClient connection
Thu, Jun 11
address feedback, remove changelog addition in favor of Max's PR
Wed, Jun 10
Looks good to me once CI passes!
Tue, Jun 9
I will close this: https://dagster.phacility.com/D3361 In favor of your PR
repin version, if works, won't require environment markers
Mon, Jun 8
I think I have some evidence that moving the list_objects operation up is fixing this. I would have to dive deeper, but I think result_for_solid() is interacting with boto and perhaps redeclaring the client. I would be comfortable merging this and then if we continue to see this flake arise, we can treat it as a p1, or disable and treat as a p2.
try re-arranging order in case boto client is being overwritten
@nate I am currently battling test flakes with Moto where occasionally it will actually make network calls to AWS and then get access denied errors on python 3.5 Funnily enough, when searching GitHub issues for Moto, I saw you mentioned here: https://github.com/spulec/moto/issues/2118
and you referenced this diff: https://dagster.phacility.com/D350
testing if unpinning moto version fixes test flake on python 3.5
Sat, Jun 6
Tests are passing now, but test_depends_on_s3_resource_intermediates seems to flake with an access denied on list objects, pretty often. I am not sure why we would get a permissions error when using mock_s3.
fix snapshot tests
fix snapshot test
fixes test errors
fixing tests to work with moto mock_S3
Fri, Jun 5
fix airline demo tests to use keyword arguments when creating s3 buckets
add moto to setup.py
removes fake s3 sessions and replaces with @mock_s3 decorator, updates docs to reflect
replaces fake_s3_resource with moto
Thu, Jun 4
Jun 3 2020
Jun 2 2020
Dictionaries in python are not guaranteed to preserve insert order before 3.6. so here I am explictly getting the account_sid and auth_token in order instead of unpacking the values. We could double splat but we need to rename the resource config keys to match Twilio's Client init's param names.
May 26 2020
- move task output assignment into seperate function
May 25 2020
- fix docker_build.sh to pass CI
I have this working with multi-step DAGs using execute plan while retrieving and storing values in the intermediate store.
I will test this with an S3 Intermediate Store. We should also figure out an architecture for allowing config data for StepInputs to be overridden by Flyte.
- removes ECR specific config
- defaultdict for compute_dict
- running on AWS KOPS cluster
- update output_name in compiler
- remove test file
May 23 2020
remove dagster_aws from setup.py
make flyte tox runs more verbose
May 22 2020
remove unnecessary intermediates_manger creation - save
ads dagit to setup.py - save
I am still testing this on multi step pipelines on Flyte infra. Expect a few more updates.
use execute plan and intermediates manager - save
May 20 2020
- This should be able to compile arbitrary DAGs into a corresponding representation in Flyte.
- This also works with solids that have StepInputs with source_types from Multiple Outputs.
- There are a few type system quirks addressed in comments.
- I have addressed the Any type feedback. However see my note on this below.
- I am planning to have the execution_plan and Intermediate store work done Wednesday night.