HomeElementl

[dagit] Repair layout worker, large DAG loading

Description

[dagit] Repair layout worker, large DAG loading

Summary:
Repair a handful of issues related to loading/rendering large DAGs in Dagit.

  • The async layout WebWorker appears to be broken in development. This looks like it is related to a dependency issue in the worker file, which is ironed out here (titleOfIO). See comment below.
  • Stop using fetch-and-network when querying for solids. These should be static anyway. There is an issue where the page gets into a query loop and never stops trying to update especially large DAGs. I'm still not quite sure what's triggering the loop, but we don't need to do more fetches after the first query anyway. (cc @alangenfeld re: our IRL discussion)
  • Refactor PipelineExplorer to make use of React.useMemo and thus avoid some unwanted extra rendering.
  • Some other implementation cleanup/changes that I made while trying to track down bugs.
  • Show a message when attempting to render a very large number of solids, based on whether we have to use the WebWorker to do so.

Test Plan:
Run dagit with -f python_modules/dagit/dagit_tests/stress/megadags.py to generate some pipelines with very large DAGs.

  • View both jobs in Dagit. Verify that there are no longer any query loops when loading the page.
  • Perform some solid subselections. Verify that the "many solids" message appears at appropriate times, and that the WebWorker completes its work and renders the DAG properly.

Reviewers: bengotow, prha, alangenfeld

Reviewed By: prha

Subscribers: alangenfeld

Differential Revision: https://dagster.phacility.com/D9124

Details

Provenance
dishAuthored on Jul 28 2021, 4:50 PM
Reviewer
prha
Differential Revision
D9124: [dagit] Repair layout worker, large DAG loading
Parents
R1:e90a7c340ea6: 0.12.4 changelog (#4417)
Branches
Unknown
Tags
Unknown