HomePhabricator

Config schema building perf improvements

Authored by schrockn on Mon, Dec 2, 11:31 PM.

Description

Config schema building perf improvements

Summary:
It turns out we were spending an extraordinary amount of time
building unused error messages while building the config type system.

Solution is not build error string in the success case.

(Next obvious thing will be to prevent create_environment_type from being called twice)

Test Plan:
See attached screenshots with profiling information.

Script used to do profiling:

from dagster import solid, pipeline, execute_pipeline

N = 10000

@solid
def return_one(_):
    #  print(f'Executing one')
    return 1

@pipeline
def big_pipeline():
    for i in range(N):
        #  print(f'Constructing {i}')
        return_one()

execute_pipeline(big_pipeline)

Before:

After:

Reviewers: max, alangenfeld

Reviewed By: alangenfeld

Differential Revision: https://dagster.phacility.com/D1528

Details

Committed
schrocknTue, Dec 3, 3:03 AM
Reviewer
alangenfeld
Differential Revision
D1528: Config schema building perf improvements
Parents
R1:86f75f7fbb17: Releasing nits
Branches
Unknown
Tags
Unknown