Page MenuHomePhabricator

Use hashed names for all system dictionaries and selectors
ClosedPublic

Authored by schrockn on Dec 6 2019, 5:19 PM.

Details

Summary

This diff eliminates names for the our system dictionary
configs. It turns out we can reuse dictionary structures and, in large
dags, this can be substantial. We are seeing 4x reducing in type
generation a commensurate improvement in config editor perf.

Test Plan

BK. Load large schema. See 4x reduction in instatiated types.

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

schrockn created this revision.Dec 6 2019, 5:19 PM
schrockn updated this revision to Diff 7167.Dec 6 2019, 6:05 PM
schrockn retitled this revision from Use hashed names for all system dictionaries to Use hashed names for all system dictionaries and selectors.
schrockn edited the summary of this revision. (Show Details)
schrockn edited the test plan for this revision. (Show Details)
schrockn added reviewers: max, alangenfeld.

upmessage

max accepted this revision.Dec 6 2019, 6:16 PM
max added inline comments.
python_modules/dagster/dagster_tests/py3_tests/test_type_examples_py3.py
590

not immediately obvious to me that this issue is related

This revision is now accepted and ready to land.Dec 6 2019, 6:16 PM
schrockn added inline comments.Dec 6 2019, 6:37 PM
python_modules/dagster/dagster_tests/py3_tests/test_type_examples_py3.py
590

nor is it to me, however it was exposed here for some reason