HomeElementl

asset catalog perf: use wipe_timestamp for asset_key filtering

Description

asset catalog perf: use wipe_timestamp for asset_key filtering

Summary:
Using SQL to filter asset keys instead of the multiple round trips to
check wipe status improves asset catalog table query perf by another 50%.

We can now just compare wipe_timestamp, last_materialization_timestamp to see if
an asset key exists (is not wiped).

Before:

Screen Shot 2021-07-13 at 4.50.17 PM.png (2×3 px, 1 MB)

After:
Screen Shot 2021-07-13 at 4.50.07 PM.png (2×3 px, 1 MB)

Test Plan: bk

Reviewers: owen, sandyryza

Reviewed By: owen

Differential Revision: https://dagster.phacility.com/D8877

Details

Provenance
prhaAuthored on Jul 13 2021, 10:40 PM
Reviewer
owen
Differential Revision
D8877: asset catalog perf: use wipe_timestamp for asset_key filtering
Parents
R1:3a8709b2e71f: mypy DagsterInstance
Branches
Unknown
Tags
Unknown