HomeElementl

[RFC][grpc] turn on gzip and set higher receive limit

Description

[RFC][grpc] turn on gzip and set higher receive limit

Summary:
The GRPC message limit is designed to prevent a client consuming unexpected memory by being pushed a large response by the server.

Due to this goal, the compression does not save us, as the limit is applied on the decompressed message.

It is still useful to enable compression as our payloads are currently filled with redundant structures that gzip eats for breakfast. The example from the test plan screen shot shows a compression down to 3% of the original message size.

Previously we have worked around this limit by creating streaming versions of the endpoints, but then we just concat all the slices together anyway so why not just raise the limit and receive the whole payload directly.

Test Plan:
integration

using dagit -f python_modules/dagit/dagit_tests/stress/megadags.py verify that message limits previously hit are no longer hit

use wireshark to inspect local traffic between

dagster api grpc -f python_modules/dagit/dagit_tests/stress/megadags.py -p 5000

and

dagit --grpc-port 5000

Screen Shot 2021-07-27 at 3.13.22 PM.png (283×817 px, 71 KB)

Reviewers: dgibson, prha, johann, rexledesma

Reviewed By: dgibson

Differential Revision: https://dagster.phacility.com/D9094

Details

Provenance
alangenfeldAuthored on Jul 27 2021, 6:59 PM
Reviewer
dgibson
Differential Revision
D9094: [RFC][grpc] turn on gzip and set higher receive limit
Parents
R1:c1cc68a27bc5: add instructions for setting up DAGSTER_HOME to new-project README
Branches
Unknown
Tags
Unknown