HomePhabricator

add scaffolding for pipeline and trip datatype

Authored by themissinghlink on Nov 25 2019, 10:33 PM.

Description

add scaffolding for pipeline and trip datatype

Summary:
This revision starts the scaffolding for the Feature pipeline and sets up a Trip DagsterType with validation checks!

Then we tack on TrafficDataTypes and WeatherDataFrame types along with a consolidated validation verification engine! With great expectations comes great responsibility. #RIPUncleBen

Test Plan:
Unit Tests and presets. If the typecheck passes then we have successfully downloaded this dataset.

Proof this works: When you execute the production preset on dagit, you should see the following!

Reviewers: max, natekupp

Reviewed By: natekupp

Differential Revision: https://dagster.phacility.com/D1461

Details

Committed
themissinghlinkNov 25 2019, 10:43 PM
Reviewer
nate
Differential Revision
D1461: add scaffolding for pipeline and trip datatype
Parents
R1:48dce58483e4: Include LICENSE in sdist and wheel (#1938)
Branches
Unknown
Tags
Unknown