Page MenuHomePhabricator

Switch subscriptions to fetch incremental updates
ClosedPublic

Authored by prha on Tue, Sep 10, 5:26 AM.

Details

Reviewers
alangenfeld
Group Reviewers
Restricted Project
Commits
R1:762ce2905dbc: Switch subscriptions to fetch incremental updates
Summary

Makes some performance optimizations to limit the amount of traffic
over the wire:

  • Adjust the polling interval
  • Change the data fetch to update the cursor (previously were fetching the full file since time of subscription)
  • Merge the incremental data on the client
Test Plan

Checked the WS traffic, saw that the payloads were smaller

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

prha created this revision.Tue, Sep 10, 5:26 AM
prha added a reviewer: Restricted Project.Wed, Sep 11, 1:24 AM
alangenfeld accepted this revision.Wed, Sep 11, 4:21 PM
alangenfeld added a subscriber: alangenfeld.
alangenfeld added inline comments.
js_modules/dagit/src/plan/ComputeLogModal.tsx
146–148

the deep nesting plus doing manipulations in line - i think this is starting to get hard to read. might just be personal opinion

python_modules/dagster/dagster/core/execution/logs.py
26

this is effectively our max chunk size?

This revision is now accepted and ready to land.Wed, Sep 11, 4:21 PM
prha updated this revision to Diff 4608.Wed, Sep 11, 6:01 PM

update with smaller chunk size, keep track of chunk-limited updates

prha updated this revision to Diff 4728.Fri, Sep 13, 6:07 PM

rebase

This revision was automatically updated to reflect the committed changes.