How to take advantage of Redis just adding it to your stack (2011) | Hacker News Reader

How to take advantage of Redis just adding it to your stack (2011)

oldblog.antirez.com

69 pointsh43k3r17 comments

If you are just starting with redis, beware that the entire dataset is held in memory, and memory is much more precious than disk. Don't just throw everything you can at it just because it's fast.

Can the db be easily distributed across machines, thus using memory from multiple machines.

ARDB[1] seems like a good choice if you want the convenience of Redis and disk persistence. It is like a Redis that's backed by Facebook's RocksDB. The redis protocol support of ARDB is pretty good.

[1]: https://github.com/yinqiwen/ardb

Here is what I always wonder about Redis:

Since it writes everything to disk, is it really faster than a SQL DB with adequate cache RAM?

When I've used Redis, I've turned disk persistence off. This is for cases where (a) consistent performance is more important than being able to recover data (like when the data expires or otherwise is used quickly, and not needed for long), and for online operations failing to a replicated slave is a more robust solution than trying to relaunch a single node and have it read data back from disk (which doesn't help you if the machine itself dies). Or, (b), the long-term persistence is handled in a separate layer that's written to separately and the Redis instances are just used for fast on-line usage (again with slaves set up to avoid losing the data from memory entirely).

You can do SQL in pure memory too, but for the use cases I've had for redis, there isn't need for a schema or firm transaction guarantees, etc, so the setup and operational simplicity of Redis wins out.

> You can do SQL in pure memory too

In MySQL, this is the MEMORY engine:

http://stackoverflow.com/questions/10692398/how-do-i-make-a-...

The InnoDB memory cache gets you pretty close as well:

http://dba.stackexchange.com/questions/11099/is-innodb-engin...

I know many of you will be well aware of how one can use redis. This is targeted for the people like me (and many others) who have just entered into this field.

The hyperloglog command set has been added to Redis since this post. Hyperloglog is useful for estimating the number of unique X over a time period w/o needing to keep track of each X. So, e.g. an easy way to estimate your daily active users.

http://antirez.com/news/75

Here's a few interesting use cases for Redis, including using it to reduce write contention to an SQL db:

http://cramer.io/2014/05/12/scaling-sql-with-redis/

The examples are in Python.

Good use case explaination for Redis. But how would you do this for a production system with 100M users. If there are over 10M users logged on at the same time you cache would be of size 10M*5000 IDS? Or would each user have an individual live cache returned per session with 5000IDS. The leaderboard example makes more sense as the top scorers number is the same throughout and the board can be updated real time.

Any time you have per user data you have an a clear shard key. i.e. instead of one Redis instance you have 100 where each serves a subset of users. In practice sharding isn't quite so simple but having a clear boundary makes it fairly easy to manage.

wow that makes sense...thanks!

You can shard client-side, but another option is twemproxy, which can shard the keyspace for you.

If you're using Python and want to shard in the client, check out nydus - https://github.com/disqus/nydus

I don't think that solution is viable within the constraints the parent post is talking about. There are many problems that come up with millions of users. Proper load balancing in that case is not simple and mostly dependent on the business logic.

What about Redis Cluster?

I used Redis in many high scale applications and it has been working very well so far.

Primarily we use it as a cache backend, and also as a temporary storage for processing data (the expire feature plays very nicely).

in a recent project I was trying to store more information in Redis (such as view counts etc..) but faced the problem of querying the data later (as it relies mainly on keys/regexp).

(2011)

Reply on news.ycombinator.com