Page MenuHomeElementl

[dagit] Lazy-load partition set info for Schedules
ClosedPublic

Authored by dish on Aug 3 2021, 7:00 PM.

Details

Summary

We currently load a ton of data for partition sets when viewing schedules, including in the "Instance status" page view of all schedules on the instance. This can be really slow, and blocks the entire query.

Separate loading partition set data into lazy subqueries that can be triggered by request from the user.

Test Plan

View /instance/schedules. Verify that the "Partition status" column has links that allow lazy-loading the partition set data. Click, verify proper loading/loaded behavior.

Verify same on Schedule permalink details.

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

dish requested review of this revision.Aug 3 2021, 7:09 PM

we can build and show the link to the partitions page with [1] right?

To me, it makes sense to

  • maybe rename column from "partition status" to "partition"
  • always show the link
  • add a "load stats" or "load status" for this expensive information
js_modules/dagit/packages/core/src/schedules/ScheduleUtils.tsx
24–25

[1]

The JS changes here look great. I'd +1 showing the partition set name all the time with a link, and then just offering to "Show Coverage" or something like that to reveal "Missing", etc.

This revision is now accepted and ready to land.Aug 4 2021, 3:11 PM

I think Bens "Show Coverage" suggestion is better the "load" ones

chefkiss