Redis 6.0 GA

Thanks, to me it means a lot that somebody out there perceives my work as something of positive :-). Live long and prosper.

IF you want to use it for caching, THEN you would use it to cache stuff for your controller.

But, Redis is much more than caching. It supports all kinda of fun data-structures like sorted lists, timeouts, sets, pub-sub and more! You can almost think of it like memory that is held by another process. In that way, there are SO many uses cases.

Well, the use for a cache is caching expensive operations. Sorry if this is just stating the obvious, but I'm not sure how else to answer how it fits into traditional MVC operations. It could be a front-end HTTP cache (although you'd probably use a CDN for that instead). It could be caching something expensive to look up or calculate, for which it's fine to not have up-to-the-second-current value.

Many people at least in Rails also use Redis to hold a queue for background processing, where it ends up working well, although hypothetically many other things could too.

You can also use redis for all sorts of other stuff that isn't caching or a queue.

Our main use case in $JOB for Redis is distributed locking. We usually do not need key-value storage and even if we do, we just go with DynamoDB instead.

STREAMS, best pubsub solution imho. I used it as a backing store for an MQTT frontend once, and also for generally coordinating worker processes to handle background tasks.

I actually use it as primary database on some parts of the mmorpg I’m developing, Redis actually has ACID capabilities so it is actually very suitable to use as primary database in gaming platforms. That being said main game server requires Mongodb due to sheer size of the data.

We use Redis to store user sessions. Spring Session Data Redis is an awesome project for this.

I always use it as my default session store. I build out large python backends and I also have been using it as a celery backend, and prefer for it to memcache for those type of tasks also.

We use redis for session storage, jobs with celery/redis-rq, view fragment caching, and certain query caching.

It can really give you a good boost in performance especially on frequently accessed pages were the content rarely changes - database queries are often "expensive" - so for frequently accessed data that doesn't change frequently - such as product descriptions - it can be a huge help.

I use it for caching, temporary content, pubsub, and distributed locking.

It's been particularly useful in load balancers / proxies for authorization, managing csrf, and tracking session activity to auto-log out users. I do this with OpenResty.

In async job or internal rpc systems, I use pubsub and streams for distributing work, ensuring exactly once processing, and getting results back to the caller.

We use it as the main database for low latency (less than 150ms) response time machine learning services. Store a pretty massive amount of data as well - close to 750GB.

As well as caching, we make really heavy use of Redis for Leaderboards for our games. The sorted sets are perfect for storing score along with the userid. Scanning the leaderboard, finding neighboring scores, etc are all really fast operations. This could probably be done with a number of other storage system but we already used Redis and we've never had a problem.

We use it to crash our servers on larger customers, by using it to cache all our user entities, pull all of them out at runtime filter them in PHP then stampede ourselves when we clear the cache

Isn't MVC a pattern for UI layers?

I'm not sure redis, a database, is relevant.

I use it as a general cache, especially for data that I know hitting the database will be expensive. I also use it for pubsub.

Use it to cache certain request payloads that are guaranteed not to change for a certain amount of time (e.g. 1-minute stock market aggregates that only change once a minute).

Redis can do far more than just caching.

However for your specific use case, considering a typical MVC web app with RDBMS data storage, you would add a check at the beginning of your Model method to return cached data if it exists, else go through the method execution and write the data to the cache just before returning it back to the controller. This way the cache would be 'warmed' on first call and data will be served directly from the cache (memory) next time till it is cleared, saving expensive disk I/O.

1. Use it for caching

2. Use it for quick lookups for user accounts

3. To queue up jobs that need to run whenever the job runner has slots available.

4. Use it to crash your entire web stack when you accidentally clear your redis instance

Really strange that how back-end developer you never had a use-case for Redis yet.

First you could check at

https://redis.io/topics/data-types-intro

If you look at single data type you can see how redis takes care about complexity(indicating the Big O notation) of each single data structure and operations.

Many devs use it for caching but in my opinion is super nice for evil-write applications.

Redis can be a very powerful tool, it can also be a sign that something has gone wrong.

Redis is a high performance key/value store with adjustable consistency. It can be used as a caching layer, and it can also do a solid job of representing a basic message queue. It typically fits in on the data layer to store computed results for fast retrieval, but it can also behave like a read replica (storing data from outside your domain).

That being said, when Redis becomes a critical part of your stack for performance reasons it usually means something has gone wrong in your data layer. I often times see teams use an antipattern of taking data from their database, changing it's shape to what they want, and then storing that new model in redis. If you're the only consumer of your data, then your data model is incorrect and you're using Redis to paper over that fact.

We use it for storing user sessions, for caching responses from a third-party API we access, and for imposing per-IP address rate limits on the use of that API. We've also previously used it for lightweight worker queues.

You need to be careful with caching inside models, because you want your models to reflect the current, completely up-to-date state of the application. Conceptually, the best place to do it is inside controllers, where you know when you can serve data which isn't completely up to date.

With skinny controllers, very often you have some specific places (eg service objects or similar layers) where the controller logic lives, and that is where you can do your caching.

For my site, I use Redis in addition to PostgreSQL for these purposes:

- User session storage

- Asynchronous "queues" for background processing using Streams (was able to eliminate RabbitMQ from my stack when I switched to this)

- Rate-limiting with a GCRA (via https://github.com/brandur/redis-cell)

- Bloom filter checks, specifically for making sure people aren't using passwords in data breaches (via https://github.com/RedisBloom/RedisBloom with data from https://haveibeenpwned.com/passwords)

Just restart your memcache server. Now you will see the point of redis.

mostly as an intermediary for pubsub architectures. ZMQ is what I've used.

For PHP applications Redis has been a must for storing sessions, especially when distributing load between multiple servers. It is also used to collect data for Prometheus exporters, for example, for languages that don't share memory between requests.

Redis is a flexible tool. Some things you could potentially do:

- This DB query is pretty slow. Cache the result in Redis.

- I'm using server-side sessions, but I have multiple servers. Where should I store session data where it'll be super fast, but all my servers can access it? You could use Redis.

- I need to do distributed locking. Use Redis

- Simple job queue? Redis.

- I'm processing a lot of inbound webhooks, and I want to avoid trying to process the same one twice. I'd like to keep a list of all webhooks I've seen in the last week, and quickly check if a new webhook is in the list, but it needs to be really fast. Redis.

Basically, Redis has support for some nifty data structures, and it's very fast. There's a lot of places in most apps where being able share a very fast list, set, map, stream, whatever between servers can be useful. Of course, all the above uses cases can be solved by other more specialised tools too, and in some cases better.

(That being said, it's so useful and generally applicable that you should be careful not to ignore fundamental issues. For example, if you have an unavoidably slow query, then by all means cache it in Redis. But if all your queries are slow because you forgot to add indexes, maybe you should fix that instead of using Redis to cache every query!)

It didn't, but we identified something that could be done without throwing away our vision. Basically Redis remains completely single threaded, but once is about to re-enter the event loop, when we have to write replies to all the clients, we use threads just for a moment in that code path, and return single threaded immediately after. It is not as performant as full threading, but we believe the right thing to do is sharding and share-nothing. Yet this 2x improvement for the single instance was almost for free so... btw it's disabled by default.

RESP3 is a strict superset of RESP2! You can just add to your implementation and end with RESP3 finally :-) It means also that a RESP3 parser can parse any RESP2 reply without issues, in case you use it against a RESP2 server.

Redis is a workhorse.

Our redis guys do that as well, but it has ran out of memory on big days. Couldn't pop out of it fast enough I guess.

Because when Redis forks to persist, every write will copy a 2MB huge page, and that's not good. All the memory will end to be copy-on-write-ed: latency spikes, big memory usage, ...

https://redis.io/topics/acl (fixed)

Looks like content is not available

Perfect!

Yep I use a non confirming release cycle and even actively backport new features form development branches to RCs or patchlevel releases if they are safe and don't have interactions with other code. This is done to make things available faster. Other times new features are almost like fixes because the old alternative resulted in user pain.

Nope - similar concept, different stories.

Agree on the Kafka, disagree on the SSL. There are compliance factors in place for certain use cases (PII).

> Now we have SSL (do people really expose Redis on the internet??)

There are no secure networks. Your options are vpn, third party ssl, or ssl in the service. Sometimes, your datacenter/cloud will guarantee "secure" network (ie: manage vpn for you).

But in many instances having ssl "inside" can be simpler.

Postgres also offers secure transport.

Security guy here. I'd argue that SSL and ACL are always good things to have, especially for systems that store data.

Modern security practices typically dictate a defense-in-depth approach. The ideas is that you will be compromised at some point (no security is perfect) and as such you should make any compromise that does happen as minimal as possible--you want to prevent attackers who get a foot in the door from rummaging around your network.

A key part of any defense-in-depth strategy are things like encryption and authentication/authorization. If you're using redis to store any kind of sensitive material, you want to make sure that only people on your network with the appropriate auth credentials can access it. This is one of the easiest ways to prevent drive-by data theft.

From here, SSL is a logical step. You need to ensure bad actors can't sniff network traffic and steal credentials.

I can't speak to streams or the other features you feel complicate Redis, but I think SSL+ACL are very important tools for increasing the cost to attackers that target redis instances leveraging those features.

I understand the sentiment, but things are a bit different than they may look. About SSL, there is no way out of this. I opposed to this feature for a long time, but simply now because of changes in regulations, policies and so forth, a lot of use cases are migrating to SSL internally even if Redis is not exposed. And frankly it is really a mess to handle SSL proxies now that everybody look like needs encryption. So what I did was the best I could do, when checking for PRs to merge: 1) Opt in, not compiled by default, no SSL libs requirements. 2) Connection abstraction, there is no SSL mentioned inside the code. Everything is in a different file.

About the "Kafka" thing, actually streams were wanted by myself, very strongly, and not suggested by Redis Labs. Let's start thinking at Redis ad a data structure server and at streams without the consumer groups part (which is totally optional). It was incredible we had no way to model a "log" in a serious way. No time series easily, using hacks and with huge memory usage because sorted sets are not the solution for this problem. But then why consumer groups? Because for a long time people had this problem of wanting a "persistent Pub/Sub": you can't lose messages just because clients disconnect in most use cases. Btw this Kafka monster is a total of 2775 lines of code, including comments. 1731 lines of code without comments. In other systems this is the prelude in the copyright notice.

But ACLs, in order to manage to survive 10 years without ACLs we had to resort to all kind of tricks: renaming commands to unguessable strings. Still with the panic of some library calling FLUSHALL for error because the developer was testing it in her/his laptop. Really ACLs have nothing to do with enterprise, but some safety is needed. The ACL monster is 1297 lines of code, and is one of the most user friendly security system you'll find in a database.

Actually all those features have a great impact on the users, huge impact on day to day operations, and are designed in order to be as simple as possible. And Redis Labs actually has only to lose from all this, because those were all potential "premium" features, instead now they are in and every other Redis provider will have it automatically as a standard. So... reality is a bit different, and it's not a conspiracy to gain market shares or alike.

SSL is a pretty important feature for almost all apps that you run in the datacenter. The idea is not to securely send Redis data to an end user on an untrusted network, the idea is to reduce the blast radius of a compromise inside your datacenter. A good example is that Slack postmortem from a couple weeks ago -- they had a proxy running inside their datacenter, and it could be convinced to make connections to internal addresses. If the service it was trying to connect to required the client to have a valid TLS certificate, the proxy would likely not provide the right credentials (because who uses client certificates on the Internet), and the connection would simply fail. A big security bug would manifest as a higher error ration in the service, instead of letting an attacker poke around in their user data. (Network based policy is also good, but is often too broad a brush. You might want the proxy to be able to talk to a database server in your network to store some results; now you can't simply add a firewall rule that says "no traffic may pass to the internal network".)

Finally, you might remember that internal NSA slideshow with the "SSL added and removed here ;-)" when talking about how they stole user data from Google's internal network. After that leak, rollout of internal mutual authentication/encryption accelerated, because people were actually inside the network observing internal RPC calls. It wasn't theoretical, it was happening.

Ultimately, mTLS is a pretty simple way to get a large increase in security. It has its disadvantages; you now have another "moving part" that can interfere with deployments and debugging (an attacker can't tcpdump, and neither can you, easily), but given how many large companies have exposed user data through unintentional interactions, it is something worth considering. It's a technique where you have to go out of your way to cause surprising behavior, and that is always good for security and reliability.

I see Redis as a toolkit that collects a number of solutions to hard distributed system problems in a single tool. It is great for developers that have a number of use cases for these kinds of things but for which there is no need or justification to spool up yet another cluster of containers/vms/servers/load balancers/etc to support it. Redis already has to do these things to be reliable and consistent; directly exposing this ability to clients and modules is a very logical thing to do. Like it or not, Redis is a platform now.

If you really just want fast data structures in memory, use memcached. If you somehow feel that Redis is a better solution for you, perhaps you should carefully consider that you may be placing more weight on its platform features than you realize.

Redis is painful to use in a highly regulated environment where all data must be encrypted in transit, all access logged and audited, etc. Personally, I think the requirements are over the top and focus on the wrong things a lot of the time. But it is what our compliance people say we must do.

We've spent hundreds of hours cobbling together a system that meets our regulatory requirements and still performs well. These features go a long way toward addressing this pain point. I think they've done a decent job making the extra complexity optional, too.

I'm a dev advocate at Redis Labs so I'm just going to reply about the last point about the protocol from my developer PoV.

Thanks to RESP3 I was able to write this client that, combined with Zig's comptime metaprogramming, can do things that no other Redis client that I've seen can do.

The user gives the desired response type and the client is able to provide appropriate translation based on what the RESP3 reply is. This would still be possible with RESP2, but v3 makes it much more robust and explicit, to the point that the ease becomes transparent without looking magical and/or triggering confusing corner cases.

https://github.com/kristoff-it/zig-okredis

On SSL, with some corporate guidelines on moving to cloud providers you must prove comms between all hosts are encrypted

> monstrous, complicated, stateful streams feature

It's two data structures (which were already in Redis for other reasons!), and an automatic sequential identifier. Everything else that's "stateful" about it is client-side state—the server is still just a data-structure server. A Redis stream is basically just a Redis sorted set that's coherent in the face of clients trying to consume it paginated as other clients insert into the middle of it.

Also, the code is in one file (https://github.com/antirez/redis/blob/unstable/src/t_stream.... ); that file is ~3KLOC. It's just another Redis Module, isolated into its own set of functions with no impact on the codebase as a whole. It's just one that's so widely applicable, to so many use-cases that people were already using Redis for (through Sidekiq/Resque/etc) that it makes sense to ship this particular module with Redis itself.

Would you get upset about bloat if Postgres upstreamed a highly-popular extension? It already has nine or ten installed by default, and a few more sitting in contrib/. But, of course, even upstreamed, none of those extensions are enabled by default, adding runtime overhead to your DB; you have to ask for them, just as if you were installing a third-party extension. Same here: if you don't use the Streams module, there's no overhead to its existence in the Redis codebase.

> do people really expose Redis on the internet??

Cloud DBaaS providers expose Redis instances "over the Internet", in the sense that they're in the same AZ but not within your VPC. To the extent that you can wireshark a data-center's virtual SDN, they need to encrypt this traffic.

Even PaaS providers do things this way, since they usually lean on third-party DBaaS providers. E.g. all of the Redis services you can attach to a Heroku app are consumed "over the Internet."

If you're using Redis through an IaaS provider's offering (e.g. AWS ElastiCache, Google Cloud Memorystore) then you get the benefit of them being able to spawn an instance "outside" your project/VPC (i.e. having it be managed by them), but have it nevertheless routed to an IP address inside your VPC. That might be enough security for you, if you don't have any legal requirements saying otherwise. For some people, it's not, and they need TLS on top anyway.

> cluster stuff

Have you looked at how it's done? It's just ease-of-use tooling around the obvious thing to do to scale Redis: partitioning the keyspace onto distinct Redis instances, and then routing requests to partitions based on the key. It's not like Redis has suddenly become a multi-master consensus system like Zookeeper; the router logic isn't even in the server codebase!

> do people really expose Redis on the internet??

You've never browsed unprotected IPs and ports huh? So many random redis instances just lying around.

Having TLS support in the main client is useful because AWS only supports AUTH if you enable TLS. Running Redis without AUTH can be kind of dangerous because Redis can kind of speak HTTP* (I think you can define custom commands to fix this) so if you have web hooks in your system and don't properly filter internal addresses then you might allow external parties to run Redis commands against your system.

* it's been years since I looked at this so maybe Redis now ships with inbuilt protection against this.

> do people really expose Redis on the internet?

Do people really still send database traffic unencrypted over unencrypted internal networks?

> Now we have SSL (do people really expose Redis on the internet??).

This is 2020. The "hard outer shell, soft chewy center" model of security is dead and it's not coming back. Modern datacenters and cloud deployments use mTLS (mutually-authenticated TLS) between every service, everywhere, all the time.

There are some massive benefits to this. For starters, you can limit what services talk to one-another entirely through distribution and signing of keys. Yes, this adds a burden of complexity if you go that route. But suddenly you don't have to care as much about (for instance) many network-exploitable vulnerabilities in your services because someone with a foothold on your network can't even get talk to your service in the first place if they don't have the right TLS cert, which is only on the handful of machines and only readable by the specific services that are legitimately allowed to connect to it.

This is a much stronger guarantee than firewalling alone (though you should also use firewalling), because multiple services can be running on a host but only the applications that are allowed to talk to your service will have read access to that key.

On the flip side, you have stronger guarantees that the service you're connecting to really is the service you're expecting it to be. If you're storing sensitive information in Redis, you can know for sure that the port you've connected to is the right Redis and not another, less-sensitive application's.

> do people really expose Redis on the internet??

Accidentally on a hobby-server, because docker-compose automatically opens up firewall ports.

Super-simple, good-enough things don't last too long. They die when the ecosystem changes, or when another super-simple player comes along and looks a bit more shiny.

It's just how the world works. You have to conquer to survive.

SSL/encrypted connections is a requirement in some regulatory frameworks. For example, transmitting PHI needs to be done over an encrypted connection.

SSL/TLS is super important. I'm glad Redis added it. Now if only Varnish would get in the game.

If Redis is already good enough for you, what actually changes for you to start disliking it? You don't have to use any of those features.

> To my thinking, Redis fit very well in the "lightweight linux thing" category.

It sounds like you don't follow Redis then.

That ship sailed years ago. Redis has at least 10 major features in addition to the caching you're talking about, including search. Redis is a kind of database now.

If you just want a cache, use memcached.

Half of my jiras at one company were related to enabling SSL for Redis due to compliance reasons (all for internal use.) Now those can be closed.

We found streams to be a breakthrough feature for pub-sub type data on IoT devices. That it can both be low-latency pub-sub and a stateful, short-lived cache is quite powerful to improve performance for many queries to the types of data generated by cameras and high-frequency sensor devices.

http://atomdocs.io/ https://github.com/elementary-robotics/atom

Personally I'm jumping up and down for ACLs. I went so far as to implement a proof of concept Redis proxy that added ACLs a couple years ago, before I heard that they would be in 6. ACLs may be niche, but when you need 'em, you need 'em!

The question would be if by adding those additional features, the experience to use the "basic/original" features got more cumbersome or the hardware requirements did change a lot. My guess it hasn't changed that much.