Page MenuHomePhabricator

RFC: Add HDFS system storage
AbandonedPublic

Authored by nate on Mar 15 2020, 7:00 PM.

Details

Summary

This works, but tests should be mocked if we are going to check it in as they currently expect live HDFS.

Test Plan

unit

Diff Detail

Repository
R1 dagster
Branch
add_hdfs
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

nate added reviewers: schrockn, alangenfeld.
nate retitled this revision from Add HDFS system storage to RFC: Add HDFS system storage.
nate edited the summary of this revision. (Show Details)
alangenfeld added a subscriber: max.

I believe @max wrote most of the storage system so is better suited to review this

Just popping back for q mgmt.

Before we move forward with this i'd like

a) to have an immediate user or
b) to have a coherent theory around labeling and organizing libraries to indicate how used/mature they are

This revision now requires changes to proceed.Mar 18 2020, 7:34 PM
nate added subscribers: sandyryza, yuhan.

Abandoning given incoming changes to this part of the system. We can revisit in the future if needed cc @sandyryza @yuhan