Page MenuHomePhabricator

Add support for using GCS for system storage
ClosedPublic

Authored by natekupp on Tue, Nov 5, 5:17 AM.

Details

Reviewers
alangenfeld
Group Reviewers
Restricted Project
Commits
R1:c992660c3a2f: Add support for using GCS for system storage
Summary

by popular* request, we can use GCS for storing things

TODO: create a GCS storage bucket for test w/ appropriate expiration policies

Test Plan

unit

Diff Detail

Repository
R1 dagster
Branch
gcs_system_storage
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

natekupp created this revision.Tue, Nov 5, 5:17 AM
natekupp updated this revision to Diff 6222.Tue, Nov 5, 5:25 AM

fix one more test

natekupp edited the summary of this revision. (Show Details)Tue, Nov 5, 5:30 AM

This is pretty cool how straightforward this was. This PR serves as pretty reasonable documentation of our system storage.

alangenfeld added inline comments.
python_modules/libraries/dagster-gcp/dagster_gcp_tests/gcs_tests/test_intermediate_store.py
177

how hard is this to address (both here and in the copy pasta source)

274

+

alangenfeld requested changes to this revision.Tue, Nov 5, 10:54 PM

to your queue to try to remove fixme's - request review if its not reasonable

This revision now requires changes to proceed.Tue, Nov 5, 10:54 PM
natekupp updated this revision to Diff 6370.Thu, Nov 7, 10:37 PM

update GCS bucket

natekupp updated this revision to Diff 6378.Thu, Nov 7, 11:09 PM

up - fix rebase issues

This revision is now accepted and ready to land.Thu, Nov 7, 11:13 PM
natekupp updated this revision to Diff 6381.Thu, Nov 7, 11:19 PM

fix pylint