HomePhabricator

change `file_handle_to_s3` solid to use S3.Client.upload_fileobj

Authored by prha on Thu, Aug 8, 6:15 PM.

Description

change file_handle_to_s3 solid to use S3.Client.upload_fileobj

Summary: Adds upload_fileobj method to the S3 resource and changes the file_handle_to_s3 solid to use it instead of put_object.

Test Plan:

  • Ran unittest (while removing a defunct one)
  • Hooked up to an actual pipeline uploading to a real S3 bucket (with creds).
  • Ran the airline demo warehouse pipeline.

Reviewers: #ft, alangenfeld

Reviewed By: #ft, alangenfeld

Subscribers: alangenfeld

Differential Revision: https://dagster.phacility.com/D784

Details

Committed
prhaThu, Aug 8, 8:51 PM
Reviewer
Restricted Project
Differential Revision
D784: change `file_handle_to_s3` solid to use S3.Client.upload_fileobj
Parents
R1:426bfa875e42: fix read-the-docs reqs.txt
Branches
Unknown
Tags
Unknown