HomePhabricator

Consolidate Baybike Pipelines

Authored by themissinghlink on Dec 9 2019, 9:08 PM.

Description

Consolidate Baybike Pipelines

Summary:
Overview

This PR consolidates the bay-bikes ETL and the ML Pipeline code into one pipeline. The reason for this was because I had a mental model mismatch and didn't understand that the intermediate system solved a lot of my problems along with the ability to run individual solids. FWIW I still need to transport files to cloud storage buckets because the weather pipeline ought to not be called over and over again since it is rate limited (and costs $$$) so that code stays.

Changelog

  • Merge ml pipeline and bay bike etl pipeline into one pipeline and opt for composite solid compositions!
  • Consolidate presets to reflect pipeline changes.
  • Deleted pipeline tests. NOTE: I fully intend to add back tests in the next revision. As written, the pipeline tests were very messy and not really adding much value. I need to rethink how to better do this because the bay bike dagster testing story needs a bit of work. In hindsight I made a mistake trying to write a pipeline integration test, I should have wrote solid tests instead (*shrug* you live and learn).

Test Plan: unit

Reviewers: nate, max, alangenfeld

Reviewed By: alangenfeld

Differential Revision: https://dagster.phacility.com/D1605

Details

Committed
themissinghlinkDec 10 2019, 4:20 PM
Reviewer
alangenfeld
Differential Revision
D1605: Consolidate Baybike Pipelines
Parents
R1:cf9373226220: Performance Improvement in core dependency structure
Branches
Unknown
Tags
Unknown