Page MenuHomePhabricator

Performance Improvement in core dependency structure
ClosedPublic

Authored by schrockn on Dec 8 2019, 6:23 AM.

Details

Summary

Profiling exposed that we were spending significant amounts of
time recomputing dependency information in the DependencyStructure class.

Turns out that this was O(N^2) w/r/t to the number of solids during a
GraphQL query that gets information about the whole graph.

Built some indexes to fix this

With this change and https://github.com/dagster-io/dagster/issues/1967 our loading
time for the config editr for large schema is now 1-2 seconds.

Test Plan

BK. Do before/after look with profiler

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

schrockn created this revision.Dec 8 2019, 6:23 AM
schrockn edited the summary of this revision. (Show Details)Dec 8 2019, 6:23 AM

Before:

After:

schrockn updated this revision to Diff 7313.Dec 8 2019, 9:08 PM
schrockn edited the summary of this revision. (Show Details)

upmessage

schrockn edited the summary of this revision. (Show Details)Dec 8 2019, 9:19 PM
schrockn updated this revision to Diff 7314.Dec 8 2019, 9:19 PM
schrockn edited the summary of this revision. (Show Details)

upmessage

schrockn updated this revision to Diff 7418.Dec 10 2019, 1:46 AM
schrockn edited the summary of this revision. (Show Details)

upmessage

This revision is now accepted and ready to land.Dec 10 2019, 4:09 PM
This revision was automatically updated to reflect the committed changes.