Thanks. I will try to understand what you're saying.
But first I'll try to explain what I think I was seeing.
Nodejs & expressjs. A couple high use REST API endpoints. Thundering herds. HTTP server can handle say max 200 rps (while maintaining target P99). Redis server can only handle sustained 100 rps.
Where does the throttle go? How is it implemented?
In my prior Java experience, I got throttling "for free" by tuning the thread pools.
With nodejs & expressjs, being single threaded with an event loop, the only solution I figured out was to throttle our app's Redis client.
What I really wanted is end-to-end backpressure. What I've done in the past is a postfix (mail server) inspired queuing system (work piled up, no blocking). What I would have settled for is an expressjs (or equiv) 'frontdoor' that throttled new socket accepts and new HTTP requests. Extra credit if the app level load balancer was aware of this frontdoor, and therefore more responsive.
If there's some app agnostic throttle buildable at layer 7, I definitely want to learn about it.