Rework Gaant for performant rendering on large DAGs, add sidebar, polish

Authored by bengotow on Feb 12 2020, 6:41 AM.


Rework Gaant for performant rendering on large DAGs, add sidebar, polish


This diff includes a ton of fixes and improvements to the Gaant viz. Most importantly it now zooms-to-fit by default, and a new sidebar shows what's executing and what's failed. Clicking a step puts it in a distinct "selection" state and the View Raw Logs and Re-Execute Step options are back.

Internally I had to almost entirely re-write the visualization to make it performant on our largest test DAG, but I'm really satisfied with the results. It's /almost/ smooth on my machine, even while the logs are streaming in. This was achieved primarily by:

  • Breaking layout into a two step process, in which we do the heavy lifting once and memoize it, and we can re-apply run metadata to adjust box sizes / colors in a fast second pass.
  • Passively observing the visible region (viewport) and clipping rendering to just boxes / lines that intersect the region.
  • Memoizing line and box rendering.


Add right hand status panel

Remove support for old EP

Colorize all three gaant viz options when metadata is available

Add single step re-execution button, persistent selection in sync with log selection

Bump blueprint and forcibly fix padding issue

Add little dots, remove labels

Split layout logic in half so metadata can be re-applied very fast

Remove scroll wheel zoom, because graph may pan both horizontally and vertically

Improve timescale label thresholds, switch slider to log scale

Adding a loading state to the logs provider / table

Fix slider cascading render, fix presets bar re-render

Observe viewport, clip offscreen boxes and make it fast 🍟

Change scale slider to β€œzoom” (0-100), map to actual px-per-ms scale in realtime for β€œfit to view” min zoom

Use DOM resizeObserver to sync viewport to panel size

Carry yellow focus into the log search box?

Put back View Raw Output button

Fix QueryInput placeholder on gaant (β€œShow Step Subset”)

Test Plan: Run snapshot tests

Reviewers: prha, schrockn, alangenfeld

Reviewed By: alangenfeld

Differential Revision: https://dagster.phacility.com/D1997