Page MenuHomePhabricator

RFC: Add HDFS system storage

Authored by nate on Mar 15 2020, 7:00 PM.



This works, but tests should be mocked if we are going to check it in as they currently expect live HDFS.

Test Plan


Diff Detail

R1 dagster
Lint OK
No Unit Test Coverage

Event Timeline

nate added reviewers: schrockn, alangenfeld.
nate retitled this revision from Add HDFS system storage to RFC: Add HDFS system storage.
nate edited the summary of this revision. (Show Details)
alangenfeld added a subscriber: max.

I believe @max wrote most of the storage system so is better suited to review this

Just popping back for q mgmt.

Before we move forward with this i'd like

a) to have an immediate user or
b) to have a coherent theory around labeling and organizing libraries to indicate how used/mature they are

This revision now requires changes to proceed.Mar 18 2020, 7:34 PM
nate added subscribers: sandyryza, yuhan.

Abandoning given incoming changes to this part of the system. We can revisit in the future if needed cc @sandyryza @yuhan