Page MenuHomePhabricator

[dagit stress] random dag gen
ClosedPublic

Authored by alangenfeld on Oct 18 2019, 5:27 PM.

Details

Reviewers
bengotow
prha
Group Reviewers
Restricted Project
Commits
R1:f838eb9a0278: [dagit stress] random dag gen
Summary

Test Plan

load dagit (slowly)

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

alangenfeld created this revision.Oct 18 2019, 5:27 PM
alangenfeld edited the summary of this revision. (Show Details)Oct 18 2019, 5:30 PM

can be tweaked and improved further but should be a good place to start for debugging

python_modules/dagit/dagit_tests/stress/megadags.py
10

easy add different variations here

prha accepted this revision.Oct 18 2019, 6:10 PM
prha added a subscriber: prha.

Probably not representative of the typical shapes we would receive, but seems like this would appropriately stress the dagit viewer.

We might want a different approach if we wanted to flex execution without having a nightmare of execution config, but can do that as a future problem

python_modules/dagit/dagit_tests/stress/dag_gen.py
34

great idea

49

clever...

This revision is now accepted and ready to land.Oct 18 2019, 6:10 PM
alangenfeld updated this revision to Diff 5901.Oct 18 2019, 6:10 PM
alangenfeld removed a reviewer: prha.

less i/o

This revision now requires review to proceed.Oct 18 2019, 6:10 PM
prha accepted this revision.Oct 29 2019, 5:44 PM

2nd time's the charm?!

This revision is now accepted and ready to land.Oct 29 2019, 5:44 PM
This revision was automatically updated to reflect the committed changes.

Oh this is super excellent, excited to do some perf work on top of this!

alangenfeld added inline comments.Oct 29 2019, 6:30 PM
python_modules/dagit/dagit_tests/stress/dag_gen.py
38–39

feel free to tweak these extremely arbitrary values as well