Hacker News Reader

Comment by umvi | Hacker News Reader

Maybe slightly off-topic but I've always struggled to understand the appeal of key-value stores like Redis and DynamoDB. I tried using it once as a substitute for shared memory in an embedded device, but you lose a lot of information (like type) and it seems like you can't represent or query complex data structures like nested structs without serializing it out to some data format first and then storing that in in the store (but at that point it's starting to get slow/complex and you can't easily examine what's in the database and so it's easier to keep using shared memory).

But clearly I'm missing something because everybody loves them and uses them. Is this a technology I would only use when building websites at scale or something?

View on HN

I was in your camp for a long time until it clicked while attending an AWS summit presentation by the head DynamoDB evangelist/wizard Rick Houlihan.

He said (paraphrasing) "a lot of people repeat this mantra that DynamoDB (and by extension, NoSQL in general) is flexible--which couldn't be further than the truth--it's not flexible, but it's extremely efficient."

He went on to elaborate that relational DBs are not going away, and if 1) your data will never be huge and 2) you just can't predict the query patterns in the future of your app--a relational DB is still the way to go.

However, if your data is both huge and you can spend the up-front careful planning of the queries you'll need to support and can afford the risk of future migration efforts that could be needed if your query needs outgrow your model--then DynamoDB is a slam dunk.

If you fall into that category, then you'll need to spend the time learning the "advanced patterns" of DynamoDB which include key overloading, adjacency-list pattern, materialized graph pattern, and hierarchical key pattern--and then compose those patterns into a custom "schema" (in the parlance of relational DBs).

I'm building an app right now and it took me about 30 hours to collectively enumerate the 40-odd queries I'll need to support and the 10 iterations of overhauls of my design. But boy, was it worth it, because the DB is going to easily be the cheapest m'fu* component of my app. Compare that to (taking an extreme alternative) paying for something like MS SQL server which is colossal money sink. Even compared to open-source like Postgres, my setup here will be probably < 1/10th the cost and if, for whatever reason, I see a drop in traffic or I need to shut down for a period, my DB isn't metering for compute--only storage.

In a drawn-out answer to your question of "would (I) only use (this) when building websites at scale or something?" the answer is generally "yep, probably".

Finely put!

I'm not a big fan of KV stores either, but I accept that they exist because there's an origin story for these things that makes a lot of sense. External storage of shared state fixes problems for shared-nothing and shared-everything programming environments.

In Java it gives teeth to the "write once, read many" pattern. By avoiding side effects you can scale your team larger. But it's a constant drain on resources to police this convention. There's always someone who thinks their reason to violate the rules is valid. Pushing the data out of process increases the cost of modifying data that should be read-only, and violations advertise loudly. You can't achieve it by obfuscation, intentional or otherwise.

Practically though, at the time KV stores were coming into existence, you could buy hardware with so much memory that Java couldn't keep up. They hit a GC wall that was causing serious problems. And running multiprocess was just anathema. If you push all of the long-lived objects out to the KV store not only does memory drop like a rock but what's left over is 'young' objects, and GC often optimizes for short-lived objects. In a way it becomes a self-fulfilling prophecy. By making in-process caching expensive they made it unwelcome.

In this same timeframe, latency on network cards became lower than latency to your drive controller. Reading something out of memory on another machine in the same data center was faster than getting it off of your own disk.

Meanwhile in an environment where all tasks are pretty isolated from each other, in-process caching is ungainly. Either you suffer with a low hit ratio, you do some sort of bizarre traffic shaping, or you do caching at the ingress point so you don't make the requests at all. But that informs your engineering priorities to such an extent most people don't like. But I see it as one of those "uncomfortable" things that Continuous Delivery advises you to face head-on. Cache invalidation is hard, yes. Wear a helmet.

So the shared-nothing people also liked these tools, which means you have two groups that probably normally wouldn't talk to each other pulling for the same team.

> In this same timeframe, latency on network cards became lower than latency to your drive controller. Reading something out of memory on another machine in the same data center was faster than getting it off of your own disk.

Was this before SSDs?

That era started before SSDs went mainstream, and it's not over. RDMA has much lower latency than NAND flash, which is why NVMe over Fabrics is getting so popular. Fetching data from a locally attached SSD isn't much faster than fetching it from a different server in the same rack.

To put some numbers on that, typical TLC flash is around 50 microseconds to access, optane and specialized SLC flash are under 10, and an infiniband connection between machines is around 1 to 1.5 microseconds.

There's a very easy to understand use-case in storing data where you don't really care about the type or querying capacity. Imagine you've got an API request coming in; your server generates the response, and then you cache that response in a TTL'd KV store where the key is some deterministic hash of the appropriate information from the request (method, path, body, requestor, etc). Future requests just use the cache for a bit.

Here, we don't care what gets stored there, because our API server isn't really "using" the data that gets returned; we just throw it back at the client. We also don't care about complex querying. That's where KV stores shine.

Its just a different pattern for different use-cases, at least in DB land. If you need complex querying, then KV probably isn't for you. But, if you need ridiculously fast lookups and perfect horizontal scalability, then you could equally say that SQL isn't for you, and KV is.

Key-value stores satisfy a set of use-cases, as do NoSQL, and SQL.

Key-value is great when you have a key-value pair that you'd like to cache. Specifically in web workloads, maybe you'd like to cache the API response (value) to a certain API query (key) such that its returned without processing power from your backend. Or maybe user permissions per action, where they don't really change as they're reliant on the user's role.

I'm not entirely sure what a valid use-case would be for an embedded device, but if there's any kind of SQL query you would make on an embedded device that returns a response that hardly changes, you might also spin up some kind of key-value store such that it's returned without (well, faster than) SQL query time.

It probably doesn't make sense to completely replace a SQL database if you're trying to represent relational data.

I've been looking at Redis recently as a potential solution to some caching I might want to do, and one hurdle I encountered almost immediately is:

Suppose I have some function, get_members_of_group(dt, groupname). It returns a set of strings. There are arbitrarily many groups, so I can't just cache all of them in one key. Someone might make a new group at any time, and groups are never deleted or changed, they just age out.

So far, this seems great. I make get_members_of_group_from_redis() first check redis for a key group_members_{dt}_{name}. If it's there, return it. If it's not, get it, cache it, return it.

But groups can also be empty.

Redis doesn't let me store an empty set. If a group comes back empty, the fact that it's empty can't be cached because you can't tell the difference between "not cached yet" and "cached but empty".

I've googled it a bit and all of the workarounds (sentinel key/value pair, storing "empty" at that key so the SMEMBERS fails with an error, etc) are hacky and make me not want to use Redis.

Is this a me problem and everyone is OK with slamming their db with hits if a cached function returns empty, or is everyone just ignoring all non-str datatypes and storing everything as JSON? Or is it just that weird to have a function that might return an empty set that nobody else has this problem?

Yes, Redis requires more manual tracking of such things. What you have to understand is that it is incredibly fast: You can easily do hundred of thousands of sismember lookups within the time it takes to serve a usual request.

> or is everyone just ignoring all non-str datatypes and storing everything as JSON

I doubt it. I've been surprised by how many of the datatypes I have found uses for.

Isolate the hack in a function with a snarky comment.

Sometimes speed is worth it.

I'd guess that most people don't see a sentinel value to distinguish between empty and unknown as a serious problem.

Supposedly you can use a Lua script - although that’s a somewhat big hammer for such a simple problem...

DynamoDB is useful for services that are likely to have a very large amount of load and require minimal maintenance. For example, Mozilla uses DynamoDB to back it's browser push notification sending service. It would be better if it supported SQL, like Google Spanner, but it's still useful.

Redis is useful because it can do so goddamn much. Caching, task queuing, hyperloglog, geospatial indexing. Set it up once, and it can replace a very large number of cloud services. And it has a lot of options for replication, backups, clustering, etc. It is like a Swiss army knife.

You're not wrong about anything.

   Is this a technology I would only use when building websites at scale or something?

Basically, though I'd amend that to read "websites that might need to scale at some point."

For a lot of projects, shared memory is a perfect choice. But if you might need to scale up to multiple instances at some point, shared memory obviously won't do. And since it's not like there's a huge penalty for using Redis, a lot of times it's just convenient to use it from the beginning.

> And since it's not like there's a huge penalty for using Redis

There is compared to shared memory.

If you actually need to scale and don't have an in-memory cache layer over Redis you're likely going to get hurt.

The real solution here is to simply write your code with an appropriate amount of abstractions, and then start with the simplest / cheapest backend that fits your needs. When it comes time to add Redis, or shared memory, or both, only a small amount of your code needs to be adjusted for the new backing store.

I find Redis is very useful when you need fast, temporary access to memory in a distributed system. I have used it several times as a distributed lock, or as storage for real-time computations in a distributed system that only use primitive values, for example aggregating page views into different clusters (hashes or sets in Redis) and then saving the result to DB.

Is Redis safe to use for a distributed lock?

If implemented correctly, there are guarantees of safety and liveness: https://redis.io/topics/distlock

The Redlock algorithm suggested for use with Redis has been the subject of some of criticism:

https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...

Kleppman is the author of Designing Data-Intensive Applications and generally someone whose opinion I trust in such things.

I love Redis and use it extensively at $dayjob, but I stick to Consul for managing distributed locks. Consul was built from the ground up to handle such things; Redis handles it as a bit more of an afterthought / consequence of other features.

True, not a native feature of Redis. I'll put Consul on my list of things to check, thanks for contributing to my knowledge.

Regarding the article, I realized I used the lock for efficiency and not correctness, and it might be why I never encountered issues.

No, it relies on time.

You should use Zookeeper, etcd or Consul

Antirez claims it is. Others are far more critical of his claims.

Personally I'd strongly suggest using a proper consensus implementation such as etcd, consul, zk.

The article mentions RocksDB, which is a great backend used in other databases. It's key value, but it can be used to build other databases, for instance graph databases.

Key-value can also be just part of a database. Like having a key-value index.

For making redis a little more usable beyond pure key-value operations take a look at their tutorial on "Secondary Indexing": https://redis.io/topics/indexes

If you care about typing and complex querying though, you might be better off using SQLite or similar.

Stack Exchange as one example aggressively uses Redis as a caching layer for practically its entire service.

Redis is very fast, dependable and relatively easy to use. The alternative, other than a competing product, is pretty much that you have to write your own custom replacement Redis in some manner. That might be fine at a small scale for fun or in an experimental product that isn't meant to go into production; otherwise you want something well proven, that many other engineers you can hire are likely to have experience with (it's hard to over-emphasize that last point).

https://meta.stackexchange.com/questions/69164/does-stack-ex...

A filesystem is a KV store; if you can solve a problem by storing data in files, then you can use a faster and more scalable KV solution.

I think it makes more sense to think of KV as a minimalistic distributed filesystem, rather than as a limited kind of memory or a stripped SQL database.

Redis lets you store and manipulate some basic data structures like linked lists, maps, and sets -- so a value in Redis doesn't necessarily have to be a big, serialized blob (though I bet that a lot of people use it this way, as if it was a drop-in replacement for Memcached).

https://redis.io/topics/data-types-intro

Redis is not a key:value store. That's memcache. Redis a data structure store with native operations that can be performed on those data structures.

I'd put it as "redis is not just a key:value store", because it certainly is one and more.

Read about:

https://www.cockroachlabs.com/blog/cockroachdb-on-rocksd/

https://blog.yugabyte.com/how-we-built-a-high-performance-do...

https://www.percona.com/blog/2018/04/30/a-look-at-myrocks-pe...

In addition to the many good answers, it's also used as a backend to SQL databases (like RocksDB which is mentioned in the article).

> but you lose a lot of information (like type)

"We have perfect support for both kinds of type: strings and and JSON blobs."

How about for storing centralized cache ? Reddis has been very helpful for us in that.

It's great when you need to share memory between pods, containers, machines, etc.

Values can be references, and you can query data structures with traversal.

Samsung didn't make this for the long-tail of consumers who will never need hardware specific solutions for their KV stores - it's for Facebook, Google, etc who build hyper-optimized KV stores that operate at huge scale.

TRUE!

Reply on news.ycombinator.com