Page MenuHomePhabricator

Config schema building perf improvements
ClosedPublic

Authored by schrockn on Mon, Dec 2, 11:41 PM.

Details

Summary

It turns out we were spending an extraordinary amount of time
building unused error messages while building the config type system.

Solution is not build error string in the success case.

(Next obvious thing will be to prevent create_environment_type from being called twice)

Test Plan

See attached screenshots with profiling information.

Script used to do profiling:

from dagster import solid, pipeline, execute_pipeline

N = 10000

@solid
def return_one(_):
    #  print(f'Executing one')
    return 1

@pipeline
def big_pipeline():
    for i in range(N):
        #  print(f'Constructing {i}')
        return_one()


execute_pipeline(big_pipeline)

Before:

After:

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

schrockn created this revision.Mon, Dec 2, 11:41 PM
schrockn retitled this revision from Schema building improvements to Config schema building perf improvements.Mon, Dec 2, 11:42 PM
schrockn edited the test plan for this revision. (Show Details)
schrockn edited the summary of this revision. (Show Details)Mon, Dec 2, 11:44 PM
schrockn edited the test plan for this revision. (Show Details)
alangenfeld accepted this revision.Tue, Dec 3, 1:25 AM

gogogo

This revision is now accepted and ready to land.Tue, Dec 3, 1:25 AM
This revision was automatically updated to reflect the committed changes.