Hacker News Reader

Comment by phowon | Hacker News Reader

As someone who is fairly tech literate but not familiar with this tech stack - in practical terms, what is Redis, and what is it used for?

View on HN

A lot of people are saying "k/v store", but I think there's an alternate definition which I've stolen from aphyr: A shared heap.

If you have a cluster of machines operating on a dataset, you can store that dataset in redis to get high performance reads and writes. In the simple case of a cache, it's a key value store. But other complex cases exist: A priority queue, an atomic transaction log, a lock server, and more.

It supports lua so if the data structure and operations you need doesn't exist you can generally build it yourself.

What is it generally used for though? To me, a "k.v" store sounds like a very generic but nice thing to have, but I still don't have a good sense of what it is and what people think of it.

The one place I've run into it is in web development where it's used for caching? In some tutorials I've read.

In a simplified manner: "I have this resource or a big piece of information I don't want to pull from my slow database or slow external service on each request/call/etc.

I'm going to see if that resource is there by doing `GET external.foo.bar`. if there is nothing here, I perform the slow pull. After the slow pull is done, we store the result in redis under the same name `external.foo.bar` with a timeout for x seconds.

Next time that resource is requested from our code, it will be there, so `GET external.foo.bar` will get us that resource without having to perform a slow call to external service. "

Thank you that was a great example. So from what I understand, Redis shouldn't be used unless there's a scaling issue?

Also, where does Redis sit (my guess is between request & database)?

Well, the previous comment was describing the use of Redis as a cache, and in that use case, yeah, it's kind of between the application and the database.

That said, caching is just one of the possible uses for Redis. I think of it as an easy way to share arrays, dictionaries, queues, etc between different applications. Then it's easy to see how it can be used for almost anything.

When used as a cache, Redis is often used as a 'look aside cache' (vs 'look thru cache'). Generally speaking, Redis does not talk directly to your database. Instead, your code looks to Redis, and then if it doesn't find what it's looking for, your code looks to your SQL database

Its commonly used to enhance the performance of web apps, ex: to put less load on your database by caching, to provide faster queries, session handling so your database doesn't has to do it, and of course, many more use cases.

Don't jump to use it unless you really have performance/scaling issues.

So lets consider one use case. We have a website where people login and we create a session to track that user across different pages on the website. Now the session will have data like user name (to display on the website), the user's location or IP address, the user's preferences, etc. So whilst the browser and the website communicate with a cookie (to identify this session), we need to store this session data somewhere. You can use a file or a database to read and write to this session data which works fine. But redis shines in this example where the speed is very good and you also have the session id as they 'key' and the session data as the 'value' that you can store.

Redis gives you amazing speed and because it provides a kv interface, the work is very easy.

Thanks, I never heard a practical example like this. I assume the "value" side of Key:Value can be a giant heap of JSON? So theoretically you can retrieve an entire collection of structured data as long as you know the key, correct?

And where the usefulness of K:V store is beaten by SQL is the point you would want to run a "JOIN" or "GROUP" on the data, for example if you wanted to count the number of keys containing a certain data point, correct?

Yes the big bag of data which is generally the value in the kv store is a json (either as a serialized string or a map if it is natively supported by redis).

Yes, the usefulness of SQL always is the join or group but with something like session, the idea is to just use it to dump values into the store which you don't want to reach into the database every time. So different teams will want to put in their own keys and data into the session object which means your DB session store becomes extremely difficult to maintain over time.

On the other hand, the joins and groups based on this can be handled later in time than in the req-response cycle itself.

The possibilities are kind of limitless. Redis implements a lot of data structures beyond a simple k/v map.

A simple non-caching example, but we use it for distributed locking, more specifically a distributed countdown latch. This is much cleaner and more performant (for us) than doing a similar operation in a traditional RDBMS.

Another very common example that is used in a lot of tutorials is maintaining a leaderboard using a sorted set.

What do you do when Redis dies on you?

All the normal strategies for dealing with distributed services still apply.

In most of these scenarios, you are still persisting data to stable storage. So, you would take the performance hit and load the data from the database.

In Rails work, two of the most common things it might be used for are: 1) A data cache, 2) a queue of work for background jobs.

This only scratches the surface of what is possible, but it's some things redis is used for.

Devil's advocate: I've seen Redis used lots but by people who mostly didn't seem to know what they're doing: NoSQL bandwagon or other magical thinking: "It's fast". Yeah, but only because it does nothing to avoid losing your data or be HA (the disk persistency and clustering options don't have pleasant semantics ATM, as far as I can tell).

You can use it as a simple cache, and that's fine. But then why not just stick with memcached -- less is more?

There are probably some scenarios where single point of failure and data loss is fine and the additional data structures redis provides over memcached are handy (e.g. analytics), but I've never seen it used for that.

See that at least makes sense to me :)

I'm not an applications programmer (and don't want to be one), so take what I say with a grain of salt, but I was first introduced to Redis several years ago during an analytical project working with a consultancy, and I asked what it was and why they wanted to bring it in.

"Its a memory-resident key-value store!"...

"So its...a hash table?"

"Its a memory resident key-value store!"...

"And don't modern languages already come with those and why aren't we just using those internal and mature solutions rather than bringing in an arbitrary new external dependency?"

blank look

"Its a memory-resident key-value store!"

The point is not the key value part, it is the "externality" and the "store" part.

Its a well optimized mature memory resident key-value store that can be used by arbitrary processes and threads which has an api accessible to arbitrary programming languages.

> "So its...a hash table?"

The answer is, yes, exactly!. Only, it's a hash table that can be shared across all of the different processes on the server. So if you have e.g. two different web requests that want to update some value, then that's how you do that. The other main alternative is a regular database, but that's much heavier and isn't really built in terms of "data structures".

(Redis isn't just a hash table - it's a list, a set, a queue, etc, in other words, all the standard library of a programming language, only in a way that can be shared across all the processes.)

What does it mean for Redis to be a message broker?

One application inserts a message into Redis, which other application(s) will read. For example, using https://redis.io/topics/pubsub

Back then, my definition was „Java collections with an API“, and to me it still fits the bill nicely. You basically have some solid computer science primitives to build whatever you want on top, only limited by memory size (which is enough for 90% of use cases)

Redis is a general purpose k/v data store that you run in memory. It has support for transactions and atomic operations, and it can be persisted and resumed. You can use it for caching, counting, event sourcing, lat/lng data, pub/sub, and more.

My coworker once called redis "NoSQLite", which I think is a very apt description.

Too much to not mention: davidw, antirez, and Richard Hipp (SQLite) are all current or former heavy hitters in the Tcl world. David contributed to the second ed. of Tcl and the Tk Toolkit[0], co-authored Rivet[1] (an Apache httpd Tcl module), among other things. antirez wrote Tcl the Misunderstood[2] and Jim[3], a lighter weight Tcl implementation, and Richard (drh) is a former Tcl Core Team member[4] who describes SQLite as “a Tcl extension that escaped into the wild.”[5]

[0] https://www.pearson.com/us/higher-education/program/Ousterho...

[1] https://tcl.apache.org/rivet/

[2] http://antirez.com/articoli/tclmisunderstood.html

[3] http://jim.tcl.tk/index.html/doc/www/www/index.html

[4] https://en.m.wikipedia.org/wiki/D._Richard_Hipp

[5] https://www.tcl.tk/community/tcl2017/assets/talk93/Paper.htm...

An out of process, memory-first, strictly consistent, distributed, data structure store.

---

Also see: https://www.dbms2.com/2008/02/18/mike-stonebraker-calls-for-...

It makes the collections you work with in your programming language (list, set, dictionary/map/hash) available as a dedicated server instead, so you can share data across multiple programs.

It's single-threaded, stores everything in RAM with optional persistence, and has Lua and modules so you can do more than the standard commands.

In memory fast key value store.

Great for caching, or anything distributed systems need quick access to.

So, like memcache?

It also supports other data structures, like lists, sets, and hashes. Super fast, and probably the best documentation I have ever seen.

Yes, exactly like memcache.

It can do things memcache can do and way more. Like storing other data types. Persisting storage to disk. Geospatial queries. Pub/sub. Lua scripting.

So, no, they are not interchangeable! Only if you just store keys and strings and do not care about persistence.

The way I like to define is: Redis is a "build your own (in memory) DB kit"

k/v store is just 10% of what it can do. Your value doesn't need to be a value, it can be:

- an array or a set

- an associative map

- an ordered list by score

- A bit array

Also offers functionality like streams and pub/sub

data-structures as a service.

need an atomic lock shared between multiple processes or servers (x)

need a set of unique values sorted by insertion time (x)

want to know the O(x) complexity of using any of the data-structures to design a system for scale (x)

need to notify multiple consumers of a modification ala pub-sub (x)

need to keep track of a stream of events for consumers (x)

need to do geo-spatial lookups (x)

oh and you want this thing to be durable to failure and easy to maintain (sentinel|cluster) (x)

redis is single process and easy to configure and understand that's my opinion of why it's so amazeballs.

[edit] unicode not supported so x instead

So you can put state that might otherwise exist on the server into Redis. This keeps your app servers stateless which has some advantages.

It's a key value store.

Where the value is not just a string but can also be a set, list, ordered set, hash (dictionary), etc.

You can store basically any common data structure in Redis and operate on individual elements as if they were local variables in your program.

A worse version of the modern hype etcd

Reply on news.ycombinator.com