The GRPC message limit is designed to prevent a client consuming unexpected memory by being pushed a large response by the server.

Due to this goal, the compression does not save us, as the limit is applied on the decompressed message.

It is still useful to enable compression as our payloads are currently filled with redundant structures that gzip eats for breakfast. The example from the test plan screen shot shows a compression down to 3% of the original message size.

Previously we have worked around this limit by creating streaming versions of the endpoints, but then we just concat all the slices together anyway so why not just raise the limit and receive the whole payload directly.

integration

using dagit -f python_modules/dagit/dagit_tests/stress/megadags.py verify that message limits previously hit are no longer hit

use wireshark to inspect local traffic between

dagster api grpc -f python_modules/dagit/dagit_tests/stress/megadags.py -p 5000

and

dagit --grpc-port 5000