Page MenuHomePhabricator

Initial dbt integration
AbandonedPublic

Authored by schrockn on Aug 10 2019, 1:48 PM.

Details

Reviewers
None
Group Reviewers
Restricted Project
Summary

Finally was able to make some time for this! This is a
prototype-quality dbt integration, but it demonstrates what the shape of
this would look like.

I copied the example from https://github.com/fishtown-analytics/jaffle_shop/

This shells out to dbt itself and runs against the database in the
examples docker container. (I had to manually create the database). It
just parses stdout with regex's, which is quite fragile.

It would be better if dbt emitted some sort of structured log so that
this could be parsed more reliably. Taylor is excited enough about this
possibility that he might tackle it. See https://github.com/fishtown-analytics/dbt/issues/1237

This emits materializations for each view or table created in the
example. This has not been thoroughly tested.

Next Steps:

  • Parse materializations out of the dbt project and render them as outputs
  • Also create a type-per-model and render metadata within dagit.
  • Consume dbt tests during a run and emit Expectations.
  • Model "seeds" as inputs into the dbt node.
Test Plan

Run Jaffle in dagit + buildkite

Diff Detail

Repository
R1 dagster
Branch
dbt-demo
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

schrockn created this revision.Aug 10 2019, 1:48 PM
schrockn updated this revision to Diff 3584.Aug 10 2019, 9:32 PM
schrockn retitled this revision from Dbt integration to Initial dbt integration.
schrockn edited the summary of this revision. (Show Details)

throw this up there

schrockn updated this revision to Diff 3585.Aug 10 2019, 9:32 PM
schrockn edited the test plan for this revision. (Show Details)
schrockn added a reviewer: Restricted Project.

reviewers

schrockn abandoned this revision.Aug 10 2019, 9:35 PM
Harbormaster failed remote builds in B2865: Diff 3584!