Mock S3 (#3241)


Mock S3 (#3241)

Previously, most of the tests in test_intermediate_storage.py failed when run locally because they relied on credentials with access to a bucket called dagster-scratch-80542c2. Similarly to #3221, I've updated the pytest fixture to instead yield a mock bucket so tests can run locally while leaving all the actual S3 implementations untouched. I also:

  • removed the nettest marks from these tests becuase they no longer cross network boundaries.
  • refactored tests slightly to use an S3 resource instead of an S3 client. I find the higher level api for resources to be more user-friendly.
  • used the real dagster_aws.s3.s3_resource instead of providing a hardcoded resource.
  • removed an unused check for AWS credentials

I didn't standardize the use of mocks outside of this directory - but I think there remains opportunity to pull this bucket fixture up a few levels to make the use of mock S3 consistent throughout dagster-aws.

If merged, #2302 will be almost to ready to close - only the ECS and EMR tests still need to be mocked.


Jordan Sanders <10291790+jmsanders@users.noreply.github.com>Authored on Nov 19 2020, 5:19 AM
GitHub <noreply@github.com>Committed on Nov 19 2020, 5:19 AM
R1:e01f57af131f: Asset View Improvements

Event Timeline

GitHub <noreply@github.com> committed R1:8b1c9f1709ad: Mock S3 (#3241) (authored by Jordan Sanders <10291790+jmsanders@users.noreply.github.com>).Nov 19 2020, 5:19 AM