Ask HN: Redis... but not in-memory? | Hacker News Reader

Ask HN: Redis... but not in-memory?

8 pointsleoh4 comments

I've grown to love redis quite a lot. Its data types are relatively easily understood, yet combinations thereof can be quite powerful. I would love to design an application using Redis' data-structure paradigms, yet I am concerned about limits regarding memory usage. Surely I could, at some interval, dump data to some other on-disk data store. Two questions then:

(1) what would be a good on-disk alternative data store?

(2) are there any alternatives to redis that are performant, yet have similar data models?

Of course, there is the VM route. But as the redis documentation itself says, "An important concept to take in mind is that Redis is not able to swap the keys, so if your memory problems are related to the fact you have too much keys with very small values, VM is not the solution."

View on HN

I was recently hit by a service running Redis exceeding RAM for the first time. I looked around and found two compatible alternatives:

https://github.com/mpalmer/redis/tree/nds-2.6

https://github.com/ideawu/ssdb

Beyond that I realized I only used get/set in my applications so I used a perl module that understands the Redis protocol and wrapped it up as a network service:

https://github.com/skx/predis

In terms of speed, writing the data to a local SQLite instance, it averages 3-4 times slower than native redis. I'm happy enough with that, although obviously I'm sure that some other projects wouldn't be.

I realize this doesn't directly address your main questions but as it is still fresh in my mind I hope it was slightly useful.

From what I know, Redis's virtual memory mode isn't bad at all. Odds are that unless you have an absurd amount of keys or have very little RAM, Redis will likely be able to fit all of the keys into memory. To help with this, use short key names, and use collection data types (so one key may be able to hold hundreds or thousands of values).

I'd say experiment with Redis, with actual testing, before looking at an alternative solution. You can set a maximum memory limit in Redis's config and see what performance is like. You can also have it prune entries once the memory limit is reached.

At the scale at which Redis stores data, RAM is pretty cheap.

I mean, nothing is really going to be as performant as Redis if you put it on disk.

The closest thing I can think of off the top of my head is MongoDB. Mongo will combine with on disk and in-memory storage. But you can do pretty much all the same things within mongo, just slower. It might not be that huge of a difference in performance depending on exactly what your doing of course. But if you're doing something in Redis that is, say, doing 100k+ ops/sec, you're highly unlikely to get that in mongo.

How much of the Redis API do you rely on? You could write a client library for say any of a dozen data stores that mimics the redis API. Disclaimer: I've only used redis for small side projects, and I'm sure some of the in-memory data structures would be difficult to replicated on disk while maintaining performance.

I use redis mostly for stream processing, keeping the latest few gigs of data in memory while streaming the old data to mongo for archiving.

Reply on news.ycombinator.com