Hacker News Reader

Comment by jzoch | Hacker News Reader

I find it really surprising that it took someone a month (I think the author stated this on Reddit) and yet redis has been so hesitant to do it. What are the downsides at this point?

View on HN

I don't think it potentially taking a long time was the primary reason for not making redis multithreaded. In the redis manifesto it is stated that one of the guiding principles is to avoid complexity if they can.

Also not that the benchmarks shown seem to be an apples and oranges comparison. If you want to compare the performance if redis vs a multithreaded fork the proper comparison involves running a redis cluster (one node per core) versus the multithreaded single node.

All else being equal I'd prefer a multithreaded approach as KeyDB is pursuing but that's just because it makes it easier to more easily utilise system resources of a single (virtual) machine.

I’m kinda against running a “cluster” on the same machine. That was a primary motivator for doing this.

It’s offloading complexity from the developers to the users which I don’t think is the right approach.

Also clusters have limitations over single instance redis. You’ll also get more Queries/GB with a Multithreaded approach.

How is a cluster more difficult for users in 2019? Most devs I know never even set up Redis. It just works through Docker/Ansible/cloud.

Have you benchmarked cluster VS locks and threads approach?

It took a month mainly because the redis codebase is so well organized. So kudos to Antirez for that. I can’t deny adding locking makes the code a bit more complicated - but there are more users than developers.

That codebase really has exceptional organization/comments.

Multithreaded code is a lot more difficult to reason about once the codebase gets large.

Redis's codebase is a little too readable compared to the complexity of the things it does. This isn't.

My experience is that it is much more a matter of architecture.

This is a lock free key value store in a single file - it doesn't have the network protocol, but any threads dealing with the network IO could use it without doing anything differently.

https://github.com/LiveAsynchronousVisualizedArchitecture/si...

antirez recently provided a full response http://antirez.com/news/126

He says most setups are network or memory bound, so I don't understand this reasoning:

"Well, that’s the plan: I/O threading is not going to happen in Redis AFAIK, because after much consideration I think it’s a lot of complexity without a good reason. Many Redis setups are network or memory bound actually. Additionally I really believe in a share-nothing setup, so the way I want to scale Redis is by improving the support for multiple Redis instances to be executed in the same host, especially via Redis Cluster."

The RAM would have to be a multiple of the dataset, because its shared-nothing, running right into one of the two common limits, right?

I would assume a cluster on a single node would partition the data, rather than replicate it. Without an in-depth knowledge of redis, I can't really predict if 4 instances sized for 1GB would have more overhead than 1 instance of 4GB, but I'm willing to hazard a guess that it would be fairly close. On the network side, it's a wash if you're already bound by the interface line rate.

This reminds me a bit of the K/kdb approach of just starting another process for almost certainly the same reasons.

Reply on news.ycombinator.com