Page MenuHomeElementl

zlib compress repos in gRPC server
AbandonedPublic

Authored by dgibson on Jul 6 2021, 10:10 PM.

Details

Summary

This is a breaking change but it's release week! zlib compression significantly lowers the size of our repos. In hindsight this was the correct move before chunking.

Test Plan

BK, manual testing confirms hacker news repo drop from 500kb to about 70kb after this change

Diff Detail

Repository
R1 dagster
Branch
zlib (branched from master)
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

there are a bunch of other chunked API methods (partitions/schedules/sensors/etc.) - could do the same thing here for those too if we want that?

Did you look in to setting compression at the grpc level for requests & responses? That could allow us to make this change at a more appropriate layer, without breaking

Also worth noting that the 4mb limit we bumped in to is something we can control. I think it applies to the raw message size, so enabling transport compression may allow us to get to a good spot to move back to sync.

python_modules/dagster/dagster/grpc/protos/api.proto
113–116

this is going to cause weird deserialization errors I suspect, should we make this protocol change in a different way? I am not familiar with navigating these changes in proto/grpc

It exists but is listed as EXPERIMENTAL in scary letters, I'll give it a try https://grpc.github.io/grpc/python/grpc.html#compression