Hacker News Reader

Comment by phamilton | Hacker News Reader

A lot of people are saying "k/v store", but I think there's an alternate definition which I've stolen from aphyr: A shared heap.

If you have a cluster of machines operating on a dataset, you can store that dataset in redis to get high performance reads and writes. In the simple case of a cache, it's a key value store. But other complex cases exist: A priority queue, an atomic transaction log, a lock server, and more.

It supports lua so if the data structure and operations you need doesn't exist you can generally build it yourself.

View on HN

See that at least makes sense to me :)

I'm not an applications programmer (and don't want to be one), so take what I say with a grain of salt, but I was first introduced to Redis several years ago during an analytical project working with a consultancy, and I asked what it was and why they wanted to bring it in.

"Its a memory-resident key-value store!"...

"So its...a hash table?"

"Its a memory resident key-value store!"...

"And don't modern languages already come with those and why aren't we just using those internal and mature solutions rather than bringing in an arbitrary new external dependency?"

blank look

"Its a memory-resident key-value store!"

The point is not the key value part, it is the "externality" and the "store" part.

Its a well optimized mature memory resident key-value store that can be used by arbitrary processes and threads which has an api accessible to arbitrary programming languages.

> "So its...a hash table?"

The answer is, yes, exactly!. Only, it's a hash table that can be shared across all of the different processes on the server. So if you have e.g. two different web requests that want to update some value, then that's how you do that. The other main alternative is a regular database, but that's much heavier and isn't really built in terms of "data structures".

(Redis isn't just a hash table - it's a list, a set, a queue, etc, in other words, all the standard library of a programming language, only in a way that can be shared across all the processes.)

Back then, my definition was „Java collections with an API“, and to me it still fits the bill nicely. You basically have some solid computer science primitives to build whatever you want on top, only limited by memory size (which is enough for 90% of use cases)

What is it generally used for though? To me, a "k.v" store sounds like a very generic but nice thing to have, but I still don't have a good sense of what it is and what people think of it.

The one place I've run into it is in web development where it's used for caching? In some tutorials I've read.

In a simplified manner: "I have this resource or a big piece of information I don't want to pull from my slow database or slow external service on each request/call/etc.

I'm going to see if that resource is there by doing `GET external.foo.bar`. if there is nothing here, I perform the slow pull. After the slow pull is done, we store the result in redis under the same name `external.foo.bar` with a timeout for x seconds.

Next time that resource is requested from our code, it will be there, so `GET external.foo.bar` will get us that resource without having to perform a slow call to external service. "

Thank you that was a great example. So from what I understand, Redis shouldn't be used unless there's a scaling issue?

Also, where does Redis sit (my guess is between request & database)?

Well, the previous comment was describing the use of Redis as a cache, and in that use case, yeah, it's kind of between the application and the database.

That said, caching is just one of the possible uses for Redis. I think of it as an easy way to share arrays, dictionaries, queues, etc between different applications. Then it's easy to see how it can be used for almost anything.

When used as a cache, Redis is often used as a 'look aside cache' (vs 'look thru cache'). Generally speaking, Redis does not talk directly to your database. Instead, your code looks to Redis, and then if it doesn't find what it's looking for, your code looks to your SQL database

Its commonly used to enhance the performance of web apps, ex: to put less load on your database by caching, to provide faster queries, session handling so your database doesn't has to do it, and of course, many more use cases.

Don't jump to use it unless you really have performance/scaling issues.

So lets consider one use case. We have a website where people login and we create a session to track that user across different pages on the website. Now the session will have data like user name (to display on the website), the user's location or IP address, the user's preferences, etc. So whilst the browser and the website communicate with a cookie (to identify this session), we need to store this session data somewhere. You can use a file or a database to read and write to this session data which works fine. But redis shines in this example where the speed is very good and you also have the session id as they 'key' and the session data as the 'value' that you can store.

Redis gives you amazing speed and because it provides a kv interface, the work is very easy.

Thanks, I never heard a practical example like this. I assume the "value" side of Key:Value can be a giant heap of JSON? So theoretically you can retrieve an entire collection of structured data as long as you know the key, correct?

And where the usefulness of K:V store is beaten by SQL is the point you would want to run a "JOIN" or "GROUP" on the data, for example if you wanted to count the number of keys containing a certain data point, correct?

Yes the big bag of data which is generally the value in the kv store is a json (either as a serialized string or a map if it is natively supported by redis).

Yes, the usefulness of SQL always is the join or group but with something like session, the idea is to just use it to dump values into the store which you don't want to reach into the database every time. So different teams will want to put in their own keys and data into the session object which means your DB session store becomes extremely difficult to maintain over time.

On the other hand, the joins and groups based on this can be handled later in time than in the req-response cycle itself.

The possibilities are kind of limitless. Redis implements a lot of data structures beyond a simple k/v map.

A simple non-caching example, but we use it for distributed locking, more specifically a distributed countdown latch. This is much cleaner and more performant (for us) than doing a similar operation in a traditional RDBMS.

Another very common example that is used in a lot of tutorials is maintaining a leaderboard using a sorted set.

What do you do when Redis dies on you?

All the normal strategies for dealing with distributed services still apply.

In most of these scenarios, you are still persisting data to stable storage. So, you would take the performance hit and load the data from the database.

In Rails work, two of the most common things it might be used for are: 1) A data cache, 2) a queue of work for background jobs.

This only scratches the surface of what is possible, but it's some things redis is used for.

Devil's advocate: I've seen Redis used lots but by people who mostly didn't seem to know what they're doing: NoSQL bandwagon or other magical thinking: "It's fast". Yeah, but only because it does nothing to avoid losing your data or be HA (the disk persistency and clustering options don't have pleasant semantics ATM, as far as I can tell).

You can use it as a simple cache, and that's fine. But then why not just stick with memcached -- less is more?

There are probably some scenarios where single point of failure and data loss is fine and the additional data structures redis provides over memcached are handy (e.g. analytics), but I've never seen it used for that.

What does it mean for Redis to be a message broker?

One application inserts a message into Redis, which other application(s) will read. For example, using https://redis.io/topics/pubsub

Reply on news.ycombinator.com