Earlier today we talked with a customer who was running 200+ pipeline runs per partition because the pipeline was re-run every 5 minutes during the day, regardless of success or failure.
While this is super unusual and our UI is designed to help you achieve one successful run per day, the UI was completely unusable for him because the query for the last 30 partitions required data from 6800 runs and took 8 MINUTES.
This is a first pass at improving the usability of this page:
- Fetches two partitions at a time, incrementally loading the charts and the matrix view
- Removes auto-refresh every 15 seconds
- Replaces some custom code in the PartitionView with pagination patterns used for Runs, etc.
I think there's a potential for this approach to backfire if the UI gets slammed with updates faster than it can re-render, but even the longitudinal pipeline streams in at a comfortable pace that the browser can handle. For very large pipelines we may see "stickiness" if React renders take 100+ms, but I propose we optimize the UI more to handle that since search, etc interactions require re-renders as well.