This is a breaking change but it's release week! zlib compression significantly lowers the size of our repos. In hindsight this was the correct move before chunking.
Details
Details
- Reviewers
alangenfeld • jordansanders • max • prha
BK, manual testing confirms hacker news repo drop from 500kb to about 70kb after this change
Diff Detail
Diff Detail
- Repository
- R1 dagster
- Branch
- zlib (branched from master)
- Lint
Lint Passed - Unit
No Test Coverage
Event Timeline
Comment Actions
there are a bunch of other chunked API methods (partitions/schedules/sensors/etc.) - could do the same thing here for those too if we want that?
Comment Actions
Did you look in to setting compression at the grpc level for requests & responses? That could allow us to make this change at a more appropriate layer, without breaking
Also worth noting that the 4mb limit we bumped in to is something we can control. I think it applies to the raw message size, so enabling transport compression may allow us to get to a good spot to move back to sync.
| python_modules/dagster/dagster/grpc/protos/api.proto | ||
|---|---|---|
| 113–116 | this is going to cause weird deserialization errors I suspect, should we make this protocol change in a different way? I am not familiar with navigating these changes in proto/grpc | |
Comment Actions
It exists but is listed as EXPERIMENTAL in scary letters, I'll give it a try https://grpc.github.io/grpc/python/grpc.html#compression