Page MenuHomePhabricator

change `file_handle_to_s3` solid to use S3.Client.upload_fileobj
ClosedPublic

Authored by prha on Thu, Aug 8, 6:49 PM.

Details

Summary

Adds upload_fileobj method to the S3 resource and changes the file_handle_to_s3 solid to use it instead of put_object.

Test Plan
  • Ran unittest (while removing a defunct one)
  • Hooked up to an actual pipeline uploading to a real S3 bucket (with creds).
  • Ran the airline demo warehouse pipeline.

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

prha created this revision.Thu, Aug 8, 6:49 PM
prha edited the summary of this revision. (Show Details)Thu, Aug 8, 6:50 PM
prha edited the test plan for this revision. (Show Details)
prha added a reviewer: Restricted Project.
alangenfeld accepted this revision.Thu, Aug 8, 7:18 PM
alangenfeld added a subscriber: alangenfeld.

thumbsup

This revision is now accepted and ready to land.Thu, Aug 8, 7:18 PM