Memcached vs. Redis – More Different Than You Would Expect
engineering.kablamo.com.au
engineering.kablamo.com.au
I see this mistake all the time, even in print! It's millis -> micros -> nanos, so...that's a lot more than a single order of magnitude.
that phrase (for countless years) feels like an attempt to sound precise, but is imprecise. great example there.
> Memcached slabs once assigned never change their size.
This is incorrect. Modern versions of Memcached has support for this. See for example [1].
[1] https://www.loginradius.com/blog/async/memcach-memory-manage...
I wrote about Pelikan, a unified cache framework, and its relationship to Memcached and Redis in this blog from 2019: https://twitter.github.io/pelikan/2019/why-pelikan.html
Specifically talked about what we would like to change about each.
I've fixed the post as well, thanks for finding this bug.
That makes me more interested in comparing the developer experience.
From a practical perspective, Redis is probably a better cache choice for a typical web developer who just wants to get things done and doesn't have major scaling/performance concerns.
It has two main advantages:
(1) Much better tooling. Memcached doesn't have a lot of support to figure out what's going on, measure hit ratio, run functions on cache data, etc. Redis has better built-in diagnostics and third-party tools and libraries.
(2) It's ideal for smaller teams to reduce infrastructure dependencies. Since Redis has many capabilities beyond in-memory caching, there's a good chance you'll be able to take advantage of it for more than just caching in the future (while it's still not really any more complicated to install/maintain than memcached).
1) memcached has excellent support for checking cache hit ratio. Just telnet to it and execute "stats". Most client libraries have support for the stats command, and there is a Prometheus exporter for memcached that works really well.
2) A problem I've seen with Redis is that it is _stateful_. While, "it's still not really any more complicated to install/maintain than memcached" is true for running it locally, I keep seeing over and over again how engineers
i) don't think about backup/restore at all.
ii) expect it to behave like a disk-backed database and don't consider it's limited to the amount of memory you have.
iii) don't consider that it (usually) is a secondary data store that can be out of sync with primary datastore (distributed commits are hard).
iv) don't consider that a proper Redis setup likely needs support for point-in-time recovery and the complexities around that and redundancy/failover.
While Redis is easy to setup as a cache and likely doesn't need the above, I keep seeing them organically grow into being used for other use cases where the above things are not in place, yielding very high operational risks sometimes not acknowledged anywhere.
Once you start doing persistence, then yes, it becomes a bit more complicated with a lot of other things to worry about, especially that you need RAM >= disk space. While it's another advantage of Redis that persistent caching is possible (and can be bolted on at a later date), taking advantage of it definitely adds some complexity and room for error.
I've seen others and myself included try to sync them when solving some problems. Just because it is so natural to believe it possible to do. Then you find out you are running circles.
Honestly for the use cases I've met Redis has only made sense to be used as a cache for performance or message bus. Your primary data store is gonna give you the other guarantees of backups, consistency etc...
Unless somebody usew it as their primary store, but everybody chooses their own religion I guess.
Lua in both places. So (ab)useful!
data structures: - yes, fewer datastructures, if any. The point isn't that they have the features or not, but that memcached is a distributed system _first_, so any feature has to make sense in that context.
"Redis is better supported, updated more often. (or maybe memcached is "finished" or has a narrower scope?)" - I've been cutting monthly releases for like 5 years now (mind the pandemic gap). Sigh.
Memory organization: This is mostly accurate but missing some major points. The sizes of the slab classes doesn't change, but slabs pages can and do get re-assigned automatically. If you assign all memory to the 1MB page class, then empty that class, memory will go back to a global pool to get re-assigned. There are edge cases but it isn't static and hasn't been for ten years.
Item size limit: The max slab size has actually been 512k internally for a long time now, despite the item limit being 1mb. Why? Because "large" items are stitched together from smaller slab chunks. Setting a 2mb or 10mb limit is fine in most use cases, but again there are edge cases, especially for very small memory limits. Usually large items aren't combined with small memory limits.
You can also _reduce the slab class overhead_ (which doesn't typically exceed 5-10%), by lowering the "slab_chunk_max" option, which puts the slab classes closer together at the expense of stitching items larger than this class. IE; if all of your objects are 16kb or less, you can freely set this limit to 16kb and reduce your slab class overhead. I'd love to make this automatic or at least reduce the defaults.
LRU: looks like the author did notice the blog post (https://memcached.org/blog/modern-lru/) - I'll add that the LRU bumping (mutex contention) is completely removed from the _access path_. This is why it scales to 48 threads. The LRU crawler is not necessary to expire items, there is also a specific thread that does the LRU balancing.
The LRU crawler is used to proactively expire items. It is highly efficient since it independently scans slab classes; the more memory an object uses the fewer neighbors it has, and it schedules when to run on each slab class, so it can "Focus" on areas with higest return.
Most of the thread scalability is pretty old; not just since 2020.
Also worth noting memcached has an efficient flash backed storage system: https://memcached.org/blog/nvm-caching/ - requires RAM to keep track of keys, but can put value data on disk. With this tradeoff we can use flash devices without burning them out, as non-get/non-set operations do not touch the SSD (ie; delete removes from memory, but doesn't cause a write). Many very huge installations of this exist.
I've also been working on an internal proxy which is nearing production-readiness for an early featureset: https://github.com/memcached/memcached/issues/827 - scriptable in lua, will have lots of useful features.
Is there a way to obtain/monitor the time stamp of LTU evictions?
I want to get a sense of how memory constrained my memcached server is, and it seems intuitive to me to monitor the “last used” date of recent evictions. Like, if The server is evicting values that haven’t been accessed in 3 months; great. But if the server is evicting values that were last used < 24 hours ago; I have concerns.
"watch evictions" command will also show a stream of details for items being evicted.
There are also wild bugs; if you google my threads on the LKML you'll find me trying to hunt down a few in the past.
Mainly what I'm doing with extstore is maintaining a clear line between what I want the OS doing and what I want the app doing: a hard rule that the memcached worker threads _cannot_ be blocked for any reason. When they submit work to extstore, they submit to background threads then return to dequeueing network traffic. If the flash disk hiccups for any reason it means some queue's can bloat but other ops may still succeed.
Further, by controlling when we defrag or drop pages we can be more careful with where writes to flash happen.
TLDR: for predictable performance. Extstore is also a lot simpler than it may sound; it's a handful of short functions built on a lot of design decisions instead of a lot of code building up an algorithm.
Partly this is because the software is so old that the thread scalability tends to track how many CPU's people actually have.
In a benchmark I made for the company I work for, Redis was only 30 - 40% faster than Postgres for our use cases because of the network trips.
Obviously I'm not familiarized with your use case, but there are a couple of things to keep in mind when performing these kind of analysis; The first one is connection cost & limit - Postgres client connection cost is quite high when compared to Redis (thats why tools like PGPool exist). Also, the upper bound connection limit in Postgres is quite low, compared to what you can achieve with Redis. Then there's the caching issue - it seems, from your description - you were actually benchmarking Postgres & filesystem cache. This happens if your dataset is small enough and your query is simple enough, but also indicates the database is idle - no other queries are running that will compete for cache resources or trigger disk I/O or buffer invalidation. If this is not your production scenario, the numbers you measured are meaningless.
Following the cache issue, by the description (and how well it fitted in cache), it seems you're not analyzing throughput. Reading 10 items is different from reading 10^6 items, and you will quickly find out that the throughput of your database server is several times lower than what you can achieve with Redis, specially considering you can easily store your data compressed. It may or may not be relevant to your particular case, but its always good to also check this.
Finally, there is the predictability aspect of the design. Having data available with predictable performance cost on a system that will happily handle tens of thousands of "simultaneous" operations, with a very low connection cost, instead of hammering your complex, multi purpose database is often a godsend, even if the performance difference was 0.
Create a replica chain and have your secondaries performing AOF writes. Page on failure.
Do an AOF rewrite or RDB dump at a regular cadence, offline, where the forking won't cause you latency.
In the past I had to use mcrouter to wrangle multiple large memcache clusters. It was too much work.
I also think that having threads helps separate points that can fail. Separate connection handling means that can slow and get jammed up but established connections can still get requests through. Slow requests don't slow down everything. Since redis does everything in the one thread stuff like evictions all stop the request handling. Its like having a gc pause in a way. also end to end encryption seems to be becoming popular in data centers and establishing those connections can be slow.
I don't think your wrong though, probably for 95% of developers redis with single thread is fine. It wouldn't be popular if it wasn't. But to be fair to myself I was asking for perfection :) At that point I dont think their would be any significant reason to pick memcache over redis.
I guess it felt like the article didn't really deliver on what it promised and instead gave something else. Which would have been fine if the topic was more clearly introduced.
> None of the the above are going to be covered in any detail. If you do need any of the above functionality, then the answer is use Redis, don't read any further and go for a walk outside.
Meanwhile Redis - used in a large codebase - seems to be tempting for developers to use it in different ways, sometimes as a cache, sometimes as a database. That in turn makes problems for devops who have to start worrying about how different redis instances are being used.
Perhaps I'm just too much of an old goat but to me, reading through all the cool features Redis has, it smells like it violates the Unix philosophy of "do one thing, well" ( https://hackaday.com/2018/09/10/doing-one-thing-well-the-uni... )