Comment by specialist | Hacker News Reader

Comment by specialist | Hacker News Reader

I'd like to see the client code, see how it manages backpressure.

(I'm probably being thick; the benchmark code is probably linked and I'm just not seeing it.)

I recently maintained some nodejs & expressjs stuff. Neither the Redis clients (or the original developers of our stuff) have any concept of backpressure (throttling). In our case, HTTP request received would cause a Redis request. For whatever reason, expressjs (or nodejs) event loop thingie processes Request preempt other stuff. So Redis responses would pile up in Redis itself, causing Redis to ABEND (out of memory).

Trying to explain backpressure, queueing, throttling to a bunch of junior devs who LOVE LOVE LOVE nodejs... Definitely was not my favorite gig.

--

PS- I started using Redis a few gigs ago. I really wanted to hate it. I'm primarily a Java dev and was esthetically offended by the NoSQL fad. But turns out Redis is awesome. And antirez is now a personal hero. I truly wish I was more like him.

> I'd like to see the client code, see how it manages backpressure.

That should (in theory) be handled on layer 7, not really on the network end (beyond regular TCP flow control), whereas this article is mostly about optimization on the network layer.

If you're fiddling with redis to the point where you're optimizing to use DPDK/OpenOnload/Exablaze/etc you've probably already exhausted the typical optimization paths (redis pipelining, tuning your queries, potentially implementing a stripped down client library implementing RESP, etc).

Thanks. I will try to understand what you're saying.

But first I'll try to explain what I think I was seeing.

Nodejs & expressjs. A couple high use REST API endpoints. Thundering herds. HTTP server can handle say max 200 rps (while maintaining target P99). Redis server can only handle sustained 100 rps.

Where does the throttle go? How is it implemented?

In my prior Java experience, I got throttling "for free" by tuning the thread pools.

With nodejs & expressjs, being single threaded with an event loop, the only solution I figured out was to throttle our app's Redis client.

What I really wanted is end-to-end backpressure. What I've done in the past is a postfix (mail server) inspired queuing system (work piled up, no blocking). What I would have settled for is an expressjs (or equiv) 'frontdoor' that throttled new socket accepts and new HTTP requests. Extra credit if the app level load balancer was aware of this frontdoor, and therefore more responsive.

If there's some app agnostic throttle buildable at layer 7, I definitely want to learn about it.

The benchmark client is memtier: https://github.com/RedisLabs/memtier_benchmark

There’s a memcached client that uses connection pools but it has its own set of problems.

And you have to use nginx plus to get the backpressure support on the ingest side. If my levels of capability and capacity were a little higher I’d be tempted to start writing a reverse proxy in Rust.

Can't you just use an atomic counter for requests/responses per connection, and increment/decrement it when you receive/send a packet, checking that it is positive before you try to send, and being fine if it briefly dips into the negative region because there shouldn't be too many fighting over being able to queue request? And even if, you should be able to find instructions or so that do a bounded-decrement that does not go below zero, and still ensures that no count are lost.

I think maybe I've seen a similar issue and didn't know it? What's the right answer for that situation, node/express hitting redis which runs out of memory?

Sorry for delay. Had to refresh my memory, dig up that old code.

I used the 'queue' (submodule?) from caolan's terrific 'async' module to limit the number of Redis requests in flight.

https://caolan.github.io/async/v3/docs.html#queue

https://www.npmjs.com/package/async

As said above, this prevents the Redis responses from piling up on the Redis server, waiting to be sent.

To troubleshoot: If the Redis CLIENT LIST's output length list, output memory, or both, are growing without ever shrinking, then I'd bet one delicious apple fritter your app's Redis client isn't processing responses fast enough. Check out the 'obl', 'oll', 'omem' stats.

https://redis.io/commands/client-list

--

Fronting Redis calls with async.queue makes the client code gnarly. So I banged out a (very minimal) client with an internal queue.

I had been weighing releasing it. And then if there's any interest, then maybe flesh out the client. There's some other notions I had wanted to try out.

I'm NOT a nodejs developer. So I hadn't fully worked out how to do proper I/O, with proper timeouts, retry, backoff. So for instance, I'd want the async.queue concurrency limit to be adaptive.

Thanks so much!

We've been running into the issue where our redis instance(s) randomly dies. I haven't been able to pin point the problem (nodejs+redis). Would love to hear your thoughts on some gotchas to look out for.

Not OP, but I wonder what the collectd plugin for Redis would tell you.

Please see (sibling?) reply to jessaustin. Thanks.

Reply on news.ycombinator.com