Page MenuHomeElementl

[dagster-dbt] re-enable asset materializations when running `dbt cli` solids
ClosedPublic

Authored by cdecarolis on May 10 2021, 4:39 PM.

Details

Summary

Fix for https://github.com/dagster-io/dagster/issues/3815.

The schema of run_results.json has changed as of DBT 0.19.0. This breaks the asset materializations in our dbt solids. This diff re-aligns the way we construct materializations such that materializations are yielded again.

Test Plan

Add to test_dbt_cli_run to make sure that the correct asset materializations are yielded when using run CLI (this was previously not covered under test). Tested fxnality on both 0.19.x and 0.18.x

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

cdecarolis retitled this revision from make node result match v1 of results.json to [dagster-dbt] make node result match v1 of results.json.May 10 2021, 4:42 PM
Harbormaster returned this revision to the author for changes because remote builds failed.May 10 2021, 5:00 PM
Harbormaster failed remote builds in B30311: Diff 37257!
Harbormaster returned this revision to the author for changes because remote builds failed.May 10 2021, 7:38 PM
Harbormaster failed remote builds in B30336: Diff 37288!

Handle difference in RPC json response

python_modules/libraries/dagster-dbt/dagster_dbt/types.py
105–114

For some reason, run result schema from dbt rpc does not match that of dbt run. This if statement attempts to account for that.

cdecarolis retitled this revision from [dagster-dbt] make node result match v1 of results.json to [dagster-dbt] re-enable asset materializations when running `dbt cli` solids.May 10 2021, 9:09 PM

It seems like this is a strict improvement over what we currently have, so we should probably merge. That said, do we have an understanding of what happened to this data that we're now missing? Did dbt stop producing it entirely? Is there a different JSON file that contains it?

It seems like this is a strict improvement over what we currently have, so we should probably merge. That said, do we have an understanding of what happened to this data that we're now missing? Did dbt stop producing it entirely? Is there a different JSON file that contains it?

This data is now contained in the [manifest.json](https://schemas.getdbt.com/dbt/manifest/v1.json) file, for the most part. This is also an output of dbt run, so it should be possible to get, but will require a restructuring of how we process this json (or maybe it just gets emitted in a different event).

respin to allow backcompat with dbt 0.18.x users

Preserve metadata for backcompat

This revision is now accepted and ready to land.Mon, May 17, 10:08 PM