> Relying on the e.g. Servlet Container not being able to handle requests seems rather bad to me. That is a very rough error handling.
It's one of those situations where the simplest "what you get by accident with a single-threaded non-evented server" solution, and the most fancy-and-complex solution, actually look alike from a client's perspective.
What you actually want is that each of your backends monitors its own resource usage, and flags itself as unhealthy in its readiness-check endpoint when it's approaching its known per-backend maximum resource capacity along any particular dimension — threads, memory usage, DB pool checked-out connections, etc. (Which can be measured quite predictably, because you're very likely running these backends in containers or VMs that enforce bounds on these resources, and then scaling the resulting predictable-consumption workload-runners horizontally.) This readiness-check failure then causes the backend to be removed from consideration as an upstream for your load-balancer / routing target for your k8s Service / etc; but existing connected flows continue to flow, gradually draining the resource consumption on that backend, until it's low enough that the backend begins reporting itself as healthy again.
Meanwhile, if the load-balancer gets a request and finds that it currently has no ready upstreams it can route to (because they're all unhealthy, because they're all at capacity) — then it responds with a 503. Just as if all those upstreams had crashed.
> Your explanations are entirely in line with mine, but I found this "messy" error handling to be exactly what I wanted to avoid.
Well, yes, but that's my point made above: this error handling is "messy" precisely because it's an encoding of user intent. It's irreducible complexity, because it's something where you want to make the decision of what to do differently in each case — e.g. a call from A to X might consider the X response critical (and so failures should be backoff-retried, and if retries exceeded, the whole job failed and rescheduled for later); while a call from B to X might consider the X response only a nice-to-have optimization over calculating the same data itself, and so it can try once, give up, and keep going.
> I made Mats first and formost not for the synchronus situation, where there is a user waiting.
I said nothing about users-as-in-humans. We're presumably both talking about a Service-Oriented Architecture here; perhaps even a microservice-oriented architecture. The "users" of Service X, above, are Service A and Service B. There's a Service X client library, that both Service A and Service B import, and make calls to Service X through. But these are still, necessarily, synchronous requests, since the further computations of Services A and B are dependent on the response from Service X.
Sure, you can queue the requests to Services A and B as long as you like; but once they're running, they're going to sit around waiting on the response from Service X (because they have nothing better to be doing while the Service X response-promise resolves.) Whether or not the Service X request is synchronous or asynchronous doesn't matter to them; they have a synchronous (though not timely) need for the data, within their own asynchronous execution.
Is this not the common pattern you see for inter-service requests within your own architecture? If not, then what is?
If what you're really talking about here is forward-only propagation of values — i.e. never needing a response (timely or not) from most of the messages you send in the first place — then you're not really talking about a messaging protocol. You're talking about a dataflow programming model, and/or a distributed CQRS/ES event store — both of which can and often are implemented on top of message queues to great effect, and neither of which purport to be sensible to use to build RPC request-response code on top of.