Page MenuHomeElementl

asset catalog perf: use wipe_timestamp for asset_key filtering
ClosedPublic

Authored by prha on Jul 13 2021, 10:43 PM.
Tags
None
Referenced Files
Unknown Object (File)
Thu, Mar 16, 4:42 AM
Unknown Object (File)
Thu, Mar 16, 2:14 AM
Unknown Object (File)
Wed, Mar 15, 10:35 PM
Unknown Object (File)
Fri, Mar 10, 2:42 PM
Unknown Object (File)
Tue, Mar 7, 12:38 AM
Unknown Object (File)
Sat, Feb 25, 7:03 PM
Unknown Object (File)
Tue, Feb 21, 9:40 PM
Unknown Object (File)
Tue, Feb 21, 6:46 PM
Subscribers
None

Details

Summary

Using SQL to filter asset keys instead of the multiple round trips to
check wipe status improves asset catalog table query perf by another 50%.

We can now just compare wipe_timestamp, last_materialization_timestamp to see if
an asset key exists (is not wiped).

Before:

Screen Shot 2021-07-13 at 4.50.17 PM.png (2×3 px, 1 MB)

After:
Screen Shot 2021-07-13 at 4.50.07 PM.png (2×3 px, 1 MB)

Test Plan

bk

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

prha edited the summary of this revision. (Show Details)
Harbormaster returned this revision to the author for changes because remote builds failed.Jul 13 2021, 11:27 PM
Harbormaster failed remote builds in B33661: Diff 41563!

fix flaky test, based on set equality

This revision is now accepted and ready to land.Jul 14 2021, 4:14 PM