Hacker News Reader

Comment by paraschopra | Hacker News Reader

I am extremely excited about Redis, however its data size limit as the amount of available RAM is a big showstopper for me. Even when the available RAM in entry level servers is getting larger and larger (however in most cases it is <512 mb for entry level VPSes) and 64 G servers are easily available, the thought that it holds all data in RAM is not very comforting. Why keep all the data (even the pieces which are seldom used) in RAM? Couldn't there be a functionality (like TC) which flushes all old data to disk from RAM and makes room in RAM for new data?

View on HN

In the mean time, I've found it's not too difficult to work around the RAM limitation. I'm working with pretty small servers, and I just use another database (MySQL usually) as a datastore for infrequently accessed data.

good point. People may wonder if this is the same as using a caching solution like memcached or using Redis itself just like a cache. It's very different for a couple of very important reasons: 1) by splitting data by different access priorities there is not to handle caching and invalidation, it's just two DBs for two kind of data, both consistent. 2) It's possible to run complex queries in the Redis side, for instance sorted lists are invaluable to take elements in order even when there are frequent score updates.

yep this is exactly the plan :) But actually it will be a good idea to use such a feature only if data access is not evenly distributed across the dataset, otherwise it's simply better to buy more RAM or more servers IMHO.

Not if you are buying SSD disks. SSD disks offer very good random read performance, so it does make sense to use them for non often accessed items.

"Very good" compared to normal disks. RAM is still much faster. Btw a setup to try is to use Redis with an SSD disk as swap, and load it with more data than RAM can hold and let the OS swap to see how it performs.

This is superficially similar to how varnish [1] handles caching: mmap it all and let the OS figure it out.

[1] http://varnish.projects.linpro.no/wiki/ArchitectNotes

IMHO, that kind of plan (flush old data to disk and keep new data) is too much work and likely to hit both performance and durability. I'd like to see redis adding more features for production ready such as this one (append only log file). The RAM usage really is not a big deal since most people have their dedicated server and can easily add more RAM.

Another question about the rewrite append only log file strategy. Will that have a possibility for loss data if the filesystem fails to recover from last power off? I mean, if you $rewrite$ a file, there is always a possibility.

Hello, actually the rewrite happens against another file and only when everything is ok it gets renamed atomically and the fd for next appends switched.

Reply on news.ycombinator.com