HomeElementl

added resource to support AWS Profiles for s3 (#3995)

Description

added resource to support AWS Profiles for s3 (#3995)

I have implemented Dagster on one of our EC2 instances. We have s3 buckets on different clusters, which use separate AWS Profiles. We needed our pipelines to be able to access appropriate s3 buckets by using these profiles. Hence, I made some small updates to dagster_aws to support the use case.

This involves:

Updating existing s3_resource to to use boto3.Session and accept profile_name as a config parameter

Appropriate changes in resources.py

Details

Provenance
deveshi <deveshi06@gmail.com>Authored on May 7 2021, 1:32 PM
GitHub <noreply@github.com>Committed on May 7 2021, 1:32 PM
Parents
R1:3c4f61c4da5c: [dagit] Add `asOf` parameter to Asset views
Branches
Unknown
Tags
Unknown

Event Timeline

GitHub <noreply@github.com> committed R1:0a81300c6fc1: added resource to support AWS Profiles for s3 (#3995) (authored by deveshi <deveshi06@gmail.com>).May 7 2021, 1:32 PM