Redis as a database (2021) | Hacker News Reader

Redis as a database (2021)

medium.com

46 pointsthunderbong39 comments

Worth noting that there is a very broad ecosystem of "Redis protocol compatible" databases that are persistence-oriented.

KVRocks: Backed by RocksDB https://github.com/apache/kvrocks

Tendis: Backed by RocksDB https://github.com/Tencent/Tendis

Tidis: Distributed, backed by TiKV https://github.com/tidb-incubator/tidis

This needs 2021 in the title, but it's actually really out of date and might as well have 2011 in the title. Redis has modules which can provide a bunch of extra capabilities that do allow querying, indexing, etc. [Full disclosure: I used to work on Redis and created much of the basis of those modules, but haven't been involved in the past 5 years]

Year added above. Thanks!

I'm not sure there's much of a reason to distinguish these cases from caching.

You're still designing the system the same way... your main considerations are still, "why/how is it OK if this thing is reset/cleared/rolled-back at any moment?" and "why/how is the data here up-to-date/recent enough for my cases?"

Serious question: why would you use Redis as a database instead of something persistence-focused with a lot of in-memory operations like MongoDB or something? Wouldn't one of those NoSQL stores like Mongo or Cassandra be the best of both worlds?

Because Redis is persistent. It's an in-memory nosql store.

It depends on the shape of your data and whether or not you need extras like pub/sub. Redis (and Redis protocol compatible databases) have rich support for HashMaps, Sets etc. Previously I've used redis-persistence as the data storage for an RBAC system (Lots of sets, lists, hashmaps, mostly flat objects).

I've also heard of it as a database for online game scoring systems.

I've come to appreciate simple APIs like the one exposed by Redis. Such simple APIs tend to require more careful planning to use but they help you to avoid running operations which are complex and inefficient. They often force you to think about your app architecture in a broader sense in order to solve problems using simpler primitives.

That said, I think the fact that Redis stores everything in memory can make it unsuitable for certain projects as a database where only a small fraction of the whole data set is ever exposed to users at any given time.

I appreciate simple APIs that can be used cross-platform, like SQL.

> simple APIs ... SQL

Which do you prefer, simple APIs, or SQL?

The biggest problem with SQL is that it was designed for humans, for writing by hand, not for programmatic access. So it suffers the same problems as shell commands do.

I also love thinking through how to store data, e.g. in Redis, and aligning it with your access patterns. The biggest challenge I’ve faced is that access patterns tend to change over time and Redis isn’t flexible (e.g. you use an HSET to pull user data but now you want to query by a different key). So you probably will end up storing data in SQL for persistence and Redis for caching, anyway.

I've also thought of redis as a distributed cache for nearly as long as I've been familiar with it. This video "I've been using Redis wrong this whole time" caught my attention recently (yup the clickbait title got me) https://www.youtube.com/watch?v=WQ61RL1GpEE and I was pleased to learn a few new things about redis on this subject

I can attest that it is certainly possible to use a Key-Value store as a database in certain conditions. I created one myself to support adding metadata (tags) to objects in the file system replacement system I was designing. Each tag had a key (the object ID it was attached to) and a value (string, number, dateTime, location, etc.).

Once I had it working and could attach hundreds of tags to each of the millions of data objects (Didgets); the whole system started to look like a sparse relational database table. So I used the tagging system to create tables and run queries against them. It turned out to be extremely flexible and fast.

To my surprise, it could often outperform traditional DB systems like SQLite or PostgreSQL. https://www.youtube.com/watch?v=Va5ZqfwQXWI

Other way around, IIRC reddit was using Postgres as a kv/nosql store.

It would be awesome if container snapshotting was good enough that taking snapshots of in-memory databases was a semi-viable option.

The demoe/paper on Quake vm's migrating across servers in particular had some cool techniques for incremental snapshotting. It d send over one image of memory, let the game-server keep running, track what was changing, then send those deltas to the other server. Trying to re-snapshot an entire in-memory database feels sizable, but for slowly changing data incremental snapshots feel like they have potential.

One of the things I've come to value as a "must have" for databases is the ability to stream CDC (Change Data Capture) out of the store.

With redis's persistance mechanisms (RDP + AOF), I see how one could set up their own CDC on top of those (tailing the AOF and reimplementing the redis instruction protocol to convert that to proper data changes), but does anyone know of a more structured way of piping out a change log?

I don't see anything obvious like a debezium connector.

You could probably use keyspace notifications: https://redis.io/docs/manual/keyspace-notifications/

But IMO, I'd be weary about data durability with Redis.

Any suggestions of tools for redis or other databases to keep state in the database synced with another system?

For example we have a service that writes that continuously does circular writes to a redis database to keep 200 megs up to date for other services to read from.

A lot of time is taken recalculating the data being updated. But its the best way we've found to eventually update even when things are going wrong.

Totally off topic, but as this is on the Wix engineering blog, is there any chance Wix will ever offer a way to deploy custom code? The last I checked (June) the only option was to use their in-browser editor. (not just talking about editing pages, but when building NodeJS extensions that run server-side)

I wonder how does the compare with dynamodb with DAX for equivalent KV store and similar throughput as a caching service. Its a lot more easier to scale to dynamo to reads and writes without worrying about key ranges etc similar to redis.

And you can also use it as a Document, Graph, Spatial, Search, Time Series and Vector DBMS like any other multi-model DB.

Databases are cool but I don’t see how this is #10 on HN.

I don't know if we have a UI to see data from a web Client & run some queries.

I suspect this will be unpopular but I'm a Redis avoider these days. I have run it in production for a few years on large projects and while it does what it says on the tin, I'd rather use another solution because having a proper schema and database engine turns out to be a really important thing on anything more than trivial. Numerous scary moments over the years which involve late nights unfucking things which would not have been issues on schema'd and carefully typed databases.

My position is that memcache is still a better cache (with a suitable static type system and serializer) and postgres is still a better database.

Sounds like you were just using Redis for something you shouldn't have been.

I think Redis is great in the space of "I need a little bit more structure than memcache", but if you're rising to the level of unscrewing a data structure or needing nontrivial types, then Redis was not the tool for that job.

I think postgres handles all those scenarios absolutely fine too.

Edit: worth noting when you under-engineer something for Redis, it's far more difficult to get it out again later than it is to add more schema to a relational DB.

Try redis-schema https://hackage.haskell.org/package/redis-schema

Absolutely no way am I going down that route when there are databases with schemas available.

Which in-memory database did you use?

Postgres + lots of RAM. In memory database is fine until you have to recover a 20 gig AOF.

We use H2 in production for a non-critical small app. If it goes down we just launch it again and it fetches data from other internal webapps.

I'd not recommend it for heavy-use in Production.

The main sell of redis is the flexibility, if you take that away then why use redis over something that treats schemas as a first class citizen?

You’re not wrong but you’ve been using redis incorrectly and were unsurprisingly happier when you used a proper rdbms.

Actually we were using it correctly according to many common patterns we were using. It was however, just bad at it compared to even a relational DB.

I love how you're so certain you're right and all of us telling you that you're wrong is just impossible for you to possibly fucking imagine.

Throw me a use case and I'll take it down. I will always put my money where my mouth is.

caching, autocomplete, or job queue.

caching = memcache. We do that.

autocomplete = inproc caching (no network turnaround). We do that.

job queue = SQS / RabbitMQ. we do that (SQS)

I didn't ask a list of alternatives you fucking wanker.

Reply on news.ycombinator.com