HomeElementl

[lakehouse] RFC: bake policies into storage defs

Description

[lakehouse] RFC: bake policies into storage defs

Summary: I got frustrated with all the code that was required to define a basic lakehouse. This simplifies the storage interface - a storage now just saves and loads stuff. For the multi-type case where we have different ways of saving / loading Spark DFs vs. Pandas DFs, there's a special storage implementation that dispatches to sub-storages.

Test Plan: bk

Reviewers: alangenfeld, schrockn

Reviewed By: alangenfeld

Differential Revision: https://dagster.phacility.com/D3883

Details

Provenance
sandyryzaAuthored on Jul 15 2020, 11:16 PM
Reviewer
alangenfeld
Differential Revision
D3883: [lakehouse] RFC: bake policies into storage defs
Parents
R1:b7d2b6e2e225: Add repository location for grpc server managed outside of dagster
Branches
Unknown
Tags
Unknown