Page MenuHomeElementl

asset catalog perf: use wipe_timestamp for asset_key filtering
ClosedPublic

Authored by prha on Jul 13 2021, 10:43 PM.

Details

Summary

Using SQL to filter asset keys instead of the multiple round trips to
check wipe status improves asset catalog table query perf by another 50%.

We can now just compare wipe_timestamp, last_materialization_timestamp to see if
an asset key exists (is not wiped).

Before:

Screen Shot 2021-07-13 at 4.50.17 PM.png (2×3 px, 1 MB)

After:
Screen Shot 2021-07-13 at 4.50.07 PM.png (2×3 px, 1 MB)

Test Plan

bk

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

prha edited the summary of this revision. (Show Details)
Harbormaster returned this revision to the author for changes because remote builds failed.Jul 13 2021, 11:27 PM
Harbormaster failed remote builds in B33661: Diff 41563!

fix flaky test, based on set equality

prha requested review of this revision.Jul 14 2021, 12:42 AM
This revision is now accepted and ready to land.Jul 14 2021, 4:14 PM