Page MenuHomePhabricator

Avoid repeated calls to readFragment to shorten new-logs critical path by 30%
ClosedPublic

Authored by bengotow on Jul 29 2019, 6:47 PM.

Details

Summary

It turns out we spend a rediculous amount of time reading the data for the pipeline
and then writing it back to the Apollo cache because their data store is heavily normalized.

I think that long term we should consider streaming the logs via a raw GraphQL fetch and not
try to store the logs into Apollo-client at all. In the very short term, we can switch from
read+write, read+write, read+write to read+write, write, write, write. Because we know that
the run data will not be updated from elsewhere.

(The highlighted segment here is what is removed by this PR)

Test Plan

Run tests

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

bengotow created this revision.Jul 29 2019, 6:47 PM
bengotow edited the summary of this revision. (Show Details)Jul 29 2019, 6:48 PM
schrockn accepted this revision.Jul 29 2019, 9:58 PM
schrockn added a subscriber: schrockn.

nice

This revision is now accepted and ready to land.Jul 29 2019, 9:58 PM