Page MenuHomePhabricator

Add support for using GCS for system storage
ClosedPublic

Authored by nate on Nov 5 2019, 5:17 AM.

Details

Reviewers
alangenfeld
Group Reviewers
Restricted Project
Commits
R1:c992660c3a2f: Add support for using GCS for system storage
Summary

by popular* request, we can use GCS for storing things

TODO: create a GCS storage bucket for test w/ appropriate expiration policies

Test Plan

unit

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

nate created this revision.Nov 5 2019, 5:17 AM
nate updated this revision to Diff 6222.Nov 5 2019, 5:25 AM

fix one more test

nate edited the summary of this revision. (Show Details)Nov 5 2019, 5:30 AM

This is pretty cool how straightforward this was. This PR serves as pretty reasonable documentation of our system storage.

alangenfeld added inline comments.
python_modules/libraries/dagster-gcp/dagster_gcp_tests/gcs_tests/test_intermediate_store.py
176

how hard is this to address (both here and in the copy pasta source)

273

+

alangenfeld requested changes to this revision.Nov 5 2019, 10:54 PM

to your queue to try to remove fixme's - request review if its not reasonable

This revision now requires changes to proceed.Nov 5 2019, 10:54 PM
nate updated this revision to Diff 6370.Nov 7 2019, 10:37 PM

update GCS bucket

nate updated this revision to Diff 6378.Nov 7 2019, 11:09 PM

up - fix rebase issues

This revision is now accepted and ready to land.Nov 7 2019, 11:13 PM
nate updated this revision to Diff 6381.Nov 7 2019, 11:19 PM

fix pylint

This revision was automatically updated to reflect the committed changes.