Page MenuHomeElementl

asset catalog perf: use wipe_timestamp for asset_key filtering
ClosedPublic

Authored by prha on Jul 13 2021, 10:43 PM.
Tags
None
Referenced Files
F2561024: D8877.diff
Sat, Sep 24, 3:59 PM
Unknown Object (File)
Tue, Sep 20, 8:54 AM
Unknown Object (File)
Sun, Sep 18, 10:55 AM
Unknown Object (File)
Fri, Sep 9, 6:50 AM
Unknown Object (File)
Sat, Sep 3, 11:31 PM
Unknown Object (File)
Fri, Sep 2, 9:35 PM
Unknown Object (File)
Sun, Aug 28, 1:28 PM
Unknown Object (File)
Fri, Aug 26, 9:07 PM
Subscribers
None

Details

Summary

Using SQL to filter asset keys instead of the multiple round trips to
check wipe status improves asset catalog table query perf by another 50%.

We can now just compare wipe_timestamp, last_materialization_timestamp to see if
an asset key exists (is not wiped).

Before:

Screen Shot 2021-07-13 at 4.50.17 PM.png (2×3 px, 1 MB)

After:
Screen Shot 2021-07-13 at 4.50.07 PM.png (2×3 px, 1 MB)

Test Plan

bk

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

prha edited the summary of this revision. (Show Details)
Harbormaster returned this revision to the author for changes because remote builds failed.Jul 13 2021, 11:27 PM
Harbormaster failed remote builds in B33661: Diff 41563!

fix flaky test, based on set equality

This revision is now accepted and ready to land.Jul 14 2021, 4:14 PM