gRPC Is Easy to Misconfigure
evanjones.ca
evanjones.ca
You probably shouldn't be sending back >8kb error messages to the client. The server should log error details locally and the logs should be inspected or scraped separately. The client doesn't need to know the full details like the contents of a variable length DS
I develop gRPC clients and servers at work that handle about 40k QPS (typically <50ms response times) whilst using default gRPC configuration. The only changes I had to make was to the max message size.
You shouldn't, but it's easy to accidentally do. Having a limit on the server and a limit on the client that's (by default) half the limit on the server seems like an unforced error - given that the server does truncate at a certain limit, surely it would be better for that limit to be smaller than the default client limit. Postel's law and all that.
While I agree, it seems like a strange limit to have as part of your spec.
As for large errors, there are plenty of cases where they can be useful. Not every RPC is crossing a privacy boundary: if an administration CLI performs an RPC that fails, it's more convenient to have lots of debugging information printed to the admin than to force them to go and dig through the server logs.
Honestly, gRPC makes me a bit sad. It's clearly a duplicate of Stubby. I don't know what Google use these days, but Stubby was a truly great RPC system. For inexplicable reasons Google constantly choose to badly reimplement their internal tech as part of open sourcing it, and gRPC is like some cheap knockoff in comparison.
If some library (that the thing being opened up uses) has an open open source version, it is necessary to worry about versioning concerns, because unlike inside google3, the open source versions can no longer make breaking changes merely by updating every single consumer. So the internal version in some cases will have drifted from the open source version. Even when a library or tool needed is already opened sourced, it is not rare for a newly open source program to use a stub version instead of the full tool/library to avoid such headaches. (e.g. I've seen super stripped down alternative to gtest shipped as part of some opened libraries, because trying to utilize the open source version was more hassle than it was worth.)
For libraries or tools not open sourced, some form of stub or adapter to some similar publicly available software is needed. In some cases that does not exist, making directly open sourcing basically impossible (but redeveloping the tool might be possible). In other cases there are alternatives available, but they might be lacking features that made the integration nicer inside Google.
Overall I can totally understand why Googlers sometimes want to basically redesign an internal tool as the approach to open sourcing the underlying concept, rather than try to publish the equivalent-ish internal tool.
I long ago concluded the actual motivating factor is that it's more fun to write new code than refactor existing code, and especially, it's more promotion worthy.
In our projects we solve this by implementing internal and external error types. External errors (e.g. form validation errors) will be sent to the client, internal errors will just send a generic error message (e.g. "please contact tech support. ID: #fa3ca....") and will be logged internally so we can trace them back to their source and ideally fix them.
This is exactly how a lot of frameworks tell you to provide secrets, and it's also how at least some cluster management tools pass secrets into apps as well. It always seemed strange to me that the environment was considered "secure".
Adding extra parts to a already complex system does not solving problems. It only transfers the problem to other components. e.g. what happens when the "scraper" also use gRPC and want to yield a >8KB error?
1) work on a team where people will make assumptions about how gRPC works because they will be wrong, for example see the article, but more generally it can be quite obtuse.
2) want to be able to easily debug over the wire information
3) expect there to be a wealth of stack overflow knowledge to be able to debug weird incompatibilities between go and JavaScript implementations
4) schema that you expect to rapidly change over time and/or versioning of said schema to avoid strange results. It’s extremely difficult to track these changes on a large codebase.
I've often thought how odd it is that the world is spending time and energy parsing things into text formats and back, when its just two computers communicating...making more work for themselves and the entire internet backbone....just because we can't be bothered to use a simple tool when debugging over the wire.
There is also a way to use one of the built in protobuf tools to do it but you also have to point at the proto files yourself.
I think there are other advantages to text protocols besides debugging on the wire, namely ease of composing tools. Streams of text is extremely versatile and economic for many workflows. Of course you lose that ease once you start embedding binary data.
I've used the gRPC keepalives intentionally for long-lived one directional streaming. You definitely need to pay careful attention to load balancing in the middle, but it does work well if you configure it properly.
If your stream messages are infrequent, you should also add the occasional Ping message in order for the server to identify and reap stale connections.
I had small ~1 core, 512mb containers handling some ~50k connections without breaking a sweat this way. The main scaling point was the goroutine stack size for each.
It seems like the correct way for the client to handle this, would be to re-establish the connection and retry if this happens. And that should be done by the grpc client library.
Does anyone have a sort of infrastructure/architecture guide at a bigger scale for gRPC?
My biggest questions range from: How do you actually load balance the servers? What happens if you have a sudden influx of requests but don't want to auto-scale? Do you still need a sort of queue-ing system in front of the gRPC server?
In my research I wasn't able to find some noteworthy articles about this and thus triggered my curiosity.
Since in my experiments I was calling a method in Php through RabbitMq from a Rust worker, so it just proved allot simpler to just use json. Also, I measured the time it took to:
1. Make a request to the Php API
2. Php sends the rpc message on the queue
3. Rust processes that message
4. Php catches the response from the worker
5. Php returns the response to the http client
It was <10ms running the cluster locally regardless if I used Protobuf or simply json.
The specific situation I saw was 1) antique perl application barfs a sql dump into msgpack 2) fluentd takes that, turns it into a thing like json but with a control character in it (think old time cyan blinky happy face or \0x03) 3) that thing gets dumped into kafka 4) logstash pulls that thing out of kafka and tries to feed it to elasticsearch 5) elasticsearch reports that it doesn't like the blinky happy face, generates an extensive log, and stores the event minus the key/value pair with the blinky happy face.
Certainly many of these steps have an implicit "don't do that!" or "Update to a newer thinger!" but the root cause (perl serializer into msgpack renders a thing that's valid msgpack but not valid json) is surprising in an unpleasant way.
I read sometime ago about about server pushback, maybe you need to override/implement your own StreamObserver to notify the client to slow down as the server isn't ready yet or configure the amount of data the gRPC server can queue there are quite a few configurations (a lot of them obscured) you can make to the server/grpc service.
In my opinion, clients themselves shouldn't worry about if the server is ready or not, only handle if the server does not respond in x seconds and then simply crash or error out. It is the same you would to with any other external service call.
I think the biggest change to gRPC is to the way I thought. Dumb clients was always something that I chose because it was an order of magnitude simpler to reason about. gRPC comes in and changes this by making the clients smart and the servers smarter, which brings allot of complexity to the table.
Also, indeed the configurations are so obscure.
I doesn't make RPC over message brokers deprecated and if that works for you under you conditions that's great.
You can say the same about message brokers. Is it LIFO, FIFO or something random? what about acknowledge, is it late or not? Can the queue hold responses (Rabbit advices against using it as a result store)?
In that regard message brokers are complicated, lets just do http calls and let the load balancer take care of routing, it's much simpler and the client is very dumb.
I suppose the problem with an easy go-to playbook is if that were easy to pick one solution for every problem, gRPC would have likely picked it by default (I can attest the team, and the external contributors, are top notch, having worked alongside them). Unfortunately some problems, at scale, need to be answered depending on architecture with the holistic system in mind, and are not just gRPC issues. Truth is large scale systems are hard to build and operate. Perhaps that is why you get help from experienced individuals and consultants.
I do agree some documentation/tooling is lacking and could be improved to guide folks through the process.
Because it supports full duplex streaming, there's a risk of tunneling your own less than fully specified protocol on top of gRPC. In some circumstances that may be worth taking advantage of, because gRPC takes care of session management, reconnecting, authorization (i.e. it has ways for you to add authorization, like headers) etc.
If you need queuing I think you should use a queue instead.
AWS Application Load Balancer is advertised to work with gRPC but we've been seeing sporadic errors. Keepalive isn't the issue, we set the IdleTimeout to the max (4000 seconds) and use server side keepalive to gc old connections.
This is horrible. Let users configure what they choose. If your network needs higher frequency keepalives, why would the authors of the spec try to know better.
I agree with the next statement:
> In my opinion, this means this setting is virtually unusable