Page MenuHomePhabricator

change `file_handle_to_s3` solid to use S3.Client.upload_fileobj

Authored by prha on Aug 8 2019, 6:49 PM.



Adds upload_fileobj method to the S3 resource and changes the file_handle_to_s3 solid to use it instead of put_object.

Test Plan
  • Ran unittest (while removing a defunct one)
  • Hooked up to an actual pipeline uploading to a real S3 bucket (with creds).
  • Ran the airline demo warehouse pipeline.

Diff Detail

R1 dagster
Automatic diff as part of commit; lint not applicable.
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

prha created this revision.Aug 8 2019, 6:49 PM
prha edited the summary of this revision. (Show Details)Aug 8 2019, 6:50 PM
prha edited the test plan for this revision. (Show Details)
prha added a reviewer: Restricted Project.
alangenfeld accepted this revision.Aug 8 2019, 7:18 PM
alangenfeld added a subscriber: alangenfeld.


This revision is now accepted and ready to land.Aug 8 2019, 7:18 PM