Hacker News Reader

Comment by jontonsoup | Hacker News Reader

Has anyone seen max (p100) client latencies of 300 to 400ms but totally normal p99? We see this across almost all our redis clusters on elasticache and have no idea why. CPU usage is tiny. Slowlog shows nothing.

View on HN

I would guess your problem is probably scheduler based. The default(ish) Linux scheduler operates in 100ms increments, the first use of a client takes 3-4 round-trips. TCP opens, block, request is sent, the client blocks on write, the client attempts to read and blocks on read. If CPU usage is high momentarily, each of these yields to another process and your client isn't scheduled for another 100ms

Hmm. We have super low CPU utilization- something like 9%. This is also across 10+ different clusters.

We also pool our clients heavily. Maybe we could reduce the new connections to zero to test.

Are you evicting or deleting large sets (or lists or sorted sets)? We use a Django ORM caching library that adds each resultset's cache key to a set of keys to invalidate when that table is updated – at which point it issues `DEL <set key>` and if that set has grown to hundreds of thousands – or millions! – of keys the main Redis process will block completely for as long as it takes to loop through and evict them.

nope!

Is the memory full and evicting? Or do you have a large db with lots of keys with ttls? Redis does a bunch of maintenance stuff on the same thread iirc in the background but not really

Memory is maybe 50% full. We are totally over provisioned. We actually just downsized and it didn’t impact anything.

We do expire but we don’t think we have a thundering herd problem with them all happening at the same time.

Is it doing backups?

My understanding is elasticache does not let you turn them off.

That would be surprising, have you tried with CONFIG SET xyz ?

Running from the client side says that the config command doesn’t exist. Not sure how to run from the server side on elasticache.

Reply on news.ycombinator.com