Simple Twitter clone using only the Redis key-value store as db and PHP | Hacker News Reader

Simple Twitter clone using only the Redis key-value store as db and PHP

code.google.com

51 pointsparaschopra15 comments

Is it true that Redis requires that the whole dataset fit in RAM?

I've seen that mentioned a few times and if so, it's a serious consideration/fact to know about when deploying apps.

Absolutely true. This is why Redis can do things that with an on-disk storage engine are almost impossible. This our reasoning: in order to really scale, with every kind of DB, if data access is evenly distributed like happens in many web 2.0 apps, the DB must fit in memory anyway (being this OS disk cache, or directly used memory like in Redis). What really matters is persistence and replication. Asynchronous disk dumps gives us all this: data is persistent, replication is fast and non blocking and trivial to setup. And because everything is in RAM we are free to do high level stuff without to care about how data is stored on disk. On disk we have just a "stupid" dump that we reload a startup.

In some way this is what Paul Graham described in some comment years ago: take all the data in memory in your application, and just write a file in append-mode to reload it later. Basically this is an ad hoc, informally-specified, bug-ridden, slow implementation of half of Redis :)

Ya but what happens when Solid state hard drives come into play. And they will be soon.

I, a asp.net guy will be able to compete with Redis then.

If RAM gets so cheap that Solid state hard drivers will come into play you'll have your Linux box with 512 GB of Ram by default. Btw I'm happy with ASP guys competing with Redis :) Also if someone stole what's good about Redis and creates FooBaredDB that is just better I'll drop Redis and use FooBaredDB.

Interesting. Thanks for the detailed explanation!

I'll be checking it out in detail (hopefully) soon.

On disk we have just a "stupid" dump that we reload a startup.

In a lazy fashion, or not? That is, if I have a 10GB data set, will I be either waiting X minutes for a basic restart or dealing with sluggish queries for a slightly lesser time?

(Update for anyone who wants an answer too: I dug around the docs and found that it's a full read into memory on restart, not lazy loading. They give an example of "It takes about 45 seconds to restore a 2 GB database on a fairly standard system, no RAID." So if frequent restarts are in your plan for some reason - not that they should be, of course - it might not be a goer.)

Redis actually looks pretty good alternative to traditional RDBMS. This tutorial really shattered quite a few myths about key-value databases' real world application. Anybody having experience with Redis? Anybody knows practical implementations of key-value dbs on a large scale?

antirez, the author, occasionally hangs out here, and I know that he's using it in production.

I actually first heard about Redis from a talk at the latest Ruby meetup in our area. The main things I took away from the talk is that it's really fast and is a pretty good way to manage sessions.

You can find more information at the Redis Google Code page:

http://code.google.com/p/redis/

Hello, there are guys using Redis in production with 350 million keys for server (64 GB ram for server). Also there are several other startups using Redis in production. Of course I'm using it in production myself.

If you search 'redis' in search.twitter.com you can see a lot of guys hacking with Redis in paywork environments. We are ready to release Redis 1.0 stable, it'a a matter of one month now. Then we'll have two development trees, one stable and one instable.

To deliver a rock solid product is one of my top commitment with Redis.

Question: (newbie here) if we assume that with 350 million keys for the 64GB ram server we have reached the peak of it's capacity, if we double the number of keys, do we have to get new hardware, like a second server or double the ram?

Or is it something simpler? Like storing the current keys on disk (somehow) and keep going with the server as it is.

To rephrase my question: keeping all in ram... is "all" like ALL ??? Would this solution work for the "real" twitter, with the billion twits, if 350 million keys were created every hour?

If you have more data than a single server can hold the best thing to do is hashing. For instance the Redis-rb client supports consistent hashing.

yes "ALL" in Ram :) this will work for twitter, facebook, everything: what they are doing is to use only the RAM even if they have MySQL, with tons of memcached around. With Redis of course the memcached layer goes away.

> If you have more data than a single server can hold the best thing to do is hashing. For instance the Redis-rb client supports consistent hashing.

Hopefully with a secret salt! It would be interesting if someone nasty managed to take down a node by e.g. predictively registering certain usernames.

The issue with key-value store isn't that you can't create an application, its that the tools to control data integrity and access your data in unexpected ways are missing or will be alot of work to create. A simple example is that two way key-value pairs for the UID and username in this article. Now any update to one value (or is it key?) must update the other.

If your main concern is scalability, then Redis and the like can look very attractive, but we should be clear on what is being given up: data coherence and query flexibility. That may be fine on your app, or even many apps, but its important to point out what is missing.

I agree in general, SQL is like magic (this is why it can't scale). You write abstract tables and then automatically you can do incredible queries against it, and this is very cool indeed.

But there are some kind of problems that Redis can fix in a very natural way. For example the idea of PUSH and LRANGE solves the get-last-N-items problem in a trivial way compared to MySQL.

Also Set intersection is able to solve tagging easily compared to the complex queries needed in a SQL db.

So SQL is very flexible, but strangely enough some trivial data access pattern is hard to model and slow even if it is as direct as take the last N items from a list.

Reply on news.ycombinator.com