Page MenuHomePhabricator

Increase gRPC message size limit for ExternalRepository API call
ClosedPublic

Authored by dgibson on Sat, Oct 10, 8:26 PM.

Details

Summary

Apparently prezi has a large enough pipeline that they are running into this. Bump the default for this specific API call since that's the one that can have large respnoses (eventually we could make it configurable in the instance too)

Test Plan

Load dagit on normal pipeslines, no noticable slowdown despite increasing the message limit
BK for large pipeline

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

sashank added inline comments.
python_modules/dagster/dagster/grpc/client.py
33

I wonder if we should bump it up this high. The default is 4194304 (4 MB) and this is 50 MB. Prezi's full pipeline comes in at about 5558113 bytes (~5.5 MB).

I guess it doesn't really matter if we plan on chunking this request soon.

python_modules/dagster/dagster_tests/api_tests/test_api_snapshot_repository.py
58–59

Nice :)

This revision is now accepted and ready to land.Sun, Oct 11, 11:34 AM

we should probably just chunk if we're going to chunk

Add a streaming/chunking API for externalrepository instead

This revision is now accepted and ready to land.Mon, Oct 12, 1:57 PM
This revision is now accepted and ready to land.Tue, Oct 13, 5:47 PM
This revision was landed with ongoing or failed builds.Thu, Oct 15, 12:48 PM
This revision was automatically updated to reflect the committed changes.