Page MenuHomeElementl

[Issue 3463] Remove versioning code paths from main execution path.
ClosedPublic

Authored by cdecarolis on Dec 21 2020, 4:32 PM.

Details

Summary

In reference to issue https://github.com/dagster-io/dagster/issues/3463.
We re-compute the execution plan at every step for the purpose of computing version information. This diff hides that computation behind an experimental tag s.t. no normal runs will have version information computed.
I verified that the version computation code paths were never hit after this change in the affected tests. This change results in a >50% improvement in the speed of test_composites.py::test_composite_basic_execution.

Test Plan

Unit tests. Verified manually that versioning code paths were never hit on a normal run.

Before change: 22s for test_composites.py::test_composite_basic_execution.
After change: 9s for test_composites.py::test_composite_basic_execution.

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

cdecarolis edited the test plan for this revision. (Show Details)
cdecarolis edited the summary of this revision. (Show Details)
Harbormaster returned this revision to the author for changes because remote builds failed.Dec 21 2020, 4:47 PM
Harbormaster failed remote builds in B23253: Diff 28274!

Since you already have the profiling set up, do you have a notion what the hot paths are for the remaining 9 seconds?

This revision is now accepted and ready to land.Dec 21 2020, 8:02 PM

Since you already have the profiling set up, do you have a notion what the hot paths are for the remaining 9 seconds?

I didn't get too great of a picture with the work I did for this, but I'll investigate further and follow up.