Page MenuHomeElementl

zlib compress repos in gRPC server

Authored by dgibson on Jul 6 2021, 10:10 PM.
Referenced Files
Unknown Object (File)
Wed, Aug 10, 4:02 AM
Unknown Object (File)
Sun, Aug 7, 2:13 PM
Unknown Object (File)
Sun, Aug 7, 2:25 AM
Unknown Object (File)
Fri, Aug 5, 11:28 AM
Unknown Object (File)
Sun, Jul 31, 4:51 AM
Unknown Object (File)
Sun, Jul 17, 11:28 PM
Unknown Object (File)
Sun, Jul 17, 8:40 PM
Unknown Object (File)
Fri, Jul 15, 3:07 AM



This is a breaking change but it's release week! zlib compression significantly lowers the size of our repos. In hindsight this was the correct move before chunking.

Test Plan

BK, manual testing confirms hacker news repo drop from 500kb to about 70kb after this change

Diff Detail

R1 dagster
zlib (branched from master)
Lint Passed
No Test Coverage

Event Timeline

there are a bunch of other chunked API methods (partitions/schedules/sensors/etc.) - could do the same thing here for those too if we want that?

Did you look in to setting compression at the grpc level for requests & responses? That could allow us to make this change at a more appropriate layer, without breaking

Also worth noting that the 4mb limit we bumped in to is something we can control. I think it applies to the raw message size, so enabling transport compression may allow us to get to a good spot to move back to sync.


this is going to cause weird deserialization errors I suspect, should we make this protocol change in a different way? I am not familiar with navigating these changes in proto/grpc

It exists but is listed as EXPERIMENTAL in scary letters, I'll give it a try