Redis 4.0 | Hacker News Reader

Redis 4.0

groups.google.com

588 pointsfs111164 comments

"9) Active memory defragmentation. Redis is able to defragment the memory while online..."

I'm so amazed that this is a thing.

Memory fragmentation is largely related to the allocator you're using (ie: glibc malloc, jemalloc, tcmalloc) and previously it was up to the OS to manage this (ie: freeing up unused memory).

Now with active memory defragmentation things are a bit more pleasant, specifically with high delete load actually freeing up unused memory in a timely manner without impacting performance too much.

Previously, to fully recover unused memory, you would have to restart the Redis server. Obviously this is not feasible but when Redis is using >50% more memory on a 120GB machine than it should then you will have to consider this an occasional housekeeping option -- now, as mentioned, this ridiculous task is no longer necessary.

120GB on a single thread, yeah. Redis has been abused for quite some time but 120GB is way out of reasonable reach for current CPU arch if you use a single core only (even if you switch off hyper threading)

Call me a slowpoke, but I wasn't aware that Redis was single core!

It makes it great for storing distributed locks, semaphores, and thanks to Lua scripting it can store/update cache invalidation lists - eg https://github.com/Suor/django-cacheops

just randomly curious here - why? hash time gets absurd?

Presumably, time complexity for most operations correspond directly with CPU usage.

See here for an example with RPUSH: https://redis.io/commands/rpush

RPUSH is O(1) which the best you can possibly get.

ZADD on the other hand is O(log(n)), which is quite good but at a large scale it becomes easier to run into performance limitations, especially on high workloads. Performing O(n) operations on large data sets is out of the question unless you're comfortable with Redis being unavailable for multiple seconds/minutes.

That said, you will have to limit yourself to operations that are computationally simple (ie: O(1) or O(log(log(n))) or pre-shard across multiple instances (or run a Redis cluster).

Redis on MOSIX might be an interesting experiment.

>Now with active memory defragmentation things are a bit more pleasant, specifically with high delete load actually freeing up unused memory in a timely manner without impacting performance too much.

Is Redis managing this itself using simple calls to Jemalloc, or is Jemalloc doing it on its own because it has better algorithms than the OS?

With regards to better algorithms than the OS -- jemalloc is not quite there yet.

Here is an illuminating discussion between Redis and Jemalloc devs regarding this:

https://github.com/jemalloc/jemalloc/issues/566

Redis will perform its own housekeeping, hence the usage of the term 'active.' AFAIK, rather than metadata being stored for this, there is a periodic active scan and manual measurement.

See more here:

https://github.com/antirez/redis/pull/3720

That's only part of the "os" if you mean part of the installed libc, as opposed to part of the kernel. Managing heap is done in userspace.

It is one of those things where people say "Java can be faster than C/C++ because it can compact memory" and the C/C++ lady says "here, hold my beer"

To be fair only a compacting GC can do that

And the Java GC is NUMA aware. The C compiler is not.

That isn't strictly true. You do not need to do garbage collection to compact memory. The data needed to do memory compaction and garbage collection largely overlaps, so you may as well do both, but you don't have to. To implement either you need to do a scan of all pointers in memory - for a garbage collection you save a currently in use list, for compaction you need to a map of where they are so you can change them latter. What you do with that data is different, but only the first step is the same. (though garbage collection can feed the empty places into the compactor which makes that work easier)

I've actually been thinking about how to do compaction in C++. The reason you cannot do it in general is code like "socket.send(new object, sizeof(void*)". The remote is then expected to send back that pointer which you cast to object and reference. Evil code for sure with a lot of failure cases that result in leaks or security holes, but possible.

Compaction is rather trivial with stop the world phase. Doing it without is a lot more challenging.

With forwarding pointers not. Which is only a problem with an FFI, which needs a traditional mark & sweep for its external callbacks anyway.

Memory fragmentation can be avoided by construction, e.g. using trees allocated in linear storage (e.g. like std::map is implemented, or the maps in libsrt [1]). For strings a binary tree (or a B/B+ could be faster), and it would not require defragmentation nor rehashing (and worst access time would be log(n), instead of O(n)). Make growing the tree using realloc() is "free" on systems with virtual memory map/remap.

[1] https://github.com/faragon/libsrt

Seriously, that sounds magical

Could anyone explain me why this is needed with virtual memory? Wouldn't the OS take care of handling gaps of free space?

@antirez: Congrats! Are you going to modularize disque now that v4 is ready?

Yes, one of the top items in the Redis 4.2 roadmap is to port Disque as a Redis module, so the steps are: 1) Implement a Redis Cluster modules API (I've already some draft design, basically there are low level stuff to enlist nodes, broadcast messages, send one-to-one messages, and have callbacks called when we receive a message), and then port Disque as Redis module on top of such API, so that we can both validate that the Cluster API is good enough with a real project, and can make Disque a first citizen of the Redis ecosystem without all the code duplication, that in the end, made the project extremely hard to sustain. Thanks.

I'm very excited about this!

I look forward to Aphyr scrutinizing it. No disrespect to you, his posts are always fun to read and show how hard it is to get this stuff correct.

If you want to try out some of the modules already available: https://redis.io/modules

Maybe with the new modules support there will emerge some explicit way to do robust worker/job queues? So you don't have to remember your BRPOPLPUSH/LREM dances (or whatever it is) just so.

Antirez has talked about moving Disque to a Redis module, which should solve that problem neatly. I'd love to drop our RabbitMQ dependency and move queueing into Redis.

Why?

one less dependency to take care of, I presume

One less service to manage, plus RabbitMQ's options for dealing with network partitions are all terrible. If you want to run a RabbitMQ cluster, you don't have any option that avoids data loss.

Can you even avoid data loss with partitions in a AP system like RabbitMQ?

RabbitMQ is not strictly an AP system. Its clustering functionality supports several different modes for handling partitions (all of which allow for data loss) and is CP. Federation, which you should use if your network between nodes isn't super reliable, is AP.[0]

What I'd like to see is clustering with support for merging data between nodes. You'd probably end up with duplicates, which I'm OK with, but you'd avoid the data loss that happens with other modes like Pause Minority, where messages get dropped on rejoining a cluster.

[0]: https://www.rabbitmq.com/distributed.html

Redis Cluster connecting to nodes via DNS instead of IP would vastly simplify deployment on K8s.

How do you solve this problem? Can you use headless K8S services for that?

You can use a StatefulSet which gives a fixed hostname per pod. Then you can have a sidecar pod that monitors the cluster, discovers the IPs of all pods, and uses redis_trib to create and maintain the cluster. It's a hassle to make it robust though.

Don't see why you would do that, instead of a headless service.

Is there a Redis PPA for Ubuntu 16.04 that is supported by the redis team?

docker images should be ready soon in case that is of interest. PR just went in - https://github.com/docker-library/redis/pull/98

Redis is of the the very few applications , that I install manually almost every time.

trick is to use the utils/install_server.sh

hey, I wrote that little script years ago, glad to hear someone (besides me) is still using it! :)

Anyone know when we might see this on AWS (Elasticache)?

Probably not soon because they use a fork of Redis.

I don't like the Elasticache offering of Redis at all. It is severely limited if you are not just using Redis as a cache. There is no way to scale up/down the size of the node without putting it down(this is possible if you run it on EC2 as well as you have access to commands related to replication that are not allowed on Elasticache). We have had master restarting instead of failing over to slave and in the process coming back with no data. Even the snapshot is only taken once in a day.

LFU policy sounds really interesting!

Thank you, there is more info in this blog post: http://antirez.com/news/109

Awesome write up. I can't wait to try out LFU with our usage. I have a suspicion that our cache hit ratio will see a nice bump.

I've said this before and I'll say it again...thank you for all you do!

The point around visualizations for measuring and tuning is an excellent one with a great example. Thanks for the link -- great writeup.

It seems it even supports nearest search for lon/lat points by default... Quite nice since most RDMS don't even support it be default.

Although I'm curious to know what algorithm it uses for nearest search, it doesn't talk about it in the doc.

I don't really understand what redis should not be used for, I guess it's not for complex queries? Conventional RDMS really seem to belong to the hard disk drive age. So the difficulty resides in having well designed data schemes.

The entire redis geo capabilities are based on geohashing.

I adore redis, I use it for tons of stuff, but I still like to keep my single-point-of-truth for critical data in an ACID, transactional relational database.

I frequently denormalize that data INTO redis so I can answer certain classes of queries quickly. I'm also very happy with redis for caching, rate limiting, inter-process-communication, distributed locking and tons of other fun use-cases.

Any news on when this will be on Azure Redis Cache ?

https://azure.microsoft.com/en-us/services/cache/

Wouldn't expect it in anything less than 3-6 months tbh... depends on how anxious MS is to git this into place.

Interesting coincidence (or maybe not...) Redis was discussed on the Floss podcast that I listened to earlier today and now I have an inkling of what Redis is. My first exposure to Redis was to ponder where it came from after I tried to run Gitlab on my puny (J1900, 4GB RAM and spinning rust) file server. It was spectacularly non-performant with most page loads timing out. I suppose it was because Redis had insufficient RAM for operation. Redis may be scalable toward large busy systems but seems less so in the other direction. I thought it would be cool to have a real Git server but this one was not it.

During the podcast the Redis guy mentioned that 4.0 was on the verge of being released.

Any reason why you lay blame on Redis and not Gitlab? Gitlab is what's in charge of serving and rendering the pages, after all.

The issues you had may have come from GitLab not having enough resources available to handle everything, as opposed to Redis itself; anecdotally, I have an application with a couple thousand keys in it that only uses ~5MiB of RAM.

Yeah, I'm in the same boat - I have a redis I use for caching auth tokens and some other values for a small service, and it doesn't take much ram at all. (5-10 MiB)

> I suppose it was because Redis had insufficient RAM for operation. Redis may be scalable toward large busy systems but seems less so in the other direction.

This is a pretty weak conclusion to draw. Did you try running Redis on its own without Gitlab? If not, I would encourage you to try it out as it takes up much less RAM than you think. The Raspberry Pi is even a supported platform...

I've said it before and I'll say it again: I am so smitten with antirez (and redis)! One of my favorite projects for sure.

Reply on news.ycombinator.com