Hacker News Reader

Comment by erulabs | Hacker News Reader

Any time these "much faster than Redis" databases come up, the sysadmin in me wonders how many people have had actual performance limitation issues with Redis. I've seen Redis servers handle hundreds of GB of traffic per hour. I've worked at companies where Aerospike and others are proposed as replacements for Redis because "they're faster" - and I point out the 98% idle CPUs on the Redis server, and the near-100%-usage CPUs on the app server fleet and mouth "But... why?"

Replacing Redis with "something faster" is a bit like removing the doors on a car because "lighter means faster!". It might look good on a racetrack, but it's about as pragmatic as climbing through a window every morning before setting off for work.

View on HN

As the Sidekiq maintainer, I’ve seen many customers need to shard Redis around 5000-10000 jobs/sec. Sharding is a major operational headache so this could be very useful to heavy job processors if it does prove to scale better.

I also find it interesting that the BSD license enables this 3rd party company to fork Redis and build closed source commercial software on top of it. One of the trade offs to consider when licensing a project.

Ok, what is a job/second?

Look up what Sidekiq is. It is awesome.

Aw, thanks! I was mis-parsing this as '5000 Redis jobs per second' instead of the intended 'needed to use sharding when sidekiq is scheduling more than 5000 jobs/second'

Thanks!

Jobs per second

May have been better to use GPL2?

Better how? If using GPLv2 means companies stay away from the project instead of building on top of it, that's not a win for anyone.

This way is a win for a company selling closed source software, that save R&D money and contributed nothing back.

As user of commercial software I am fine with it, not so sure if FOSS advocates at large will be so happy when only non-copyleft licenses survive and we are back in the shareware/pd libraries days.

> and contributed nothing back

Why do we assume closed-source software vendors contribute nothing back?

Speaking as an employee at a company that produces a closed-source software product that uses open-source libraries, I've contributed plenty back to various libraries, including publishing some of my own.

Libraries with permissive licenses get more users, and more users mean more opportunities for receiving contributions. Something like GPLv2 is really only truly effective at soliciting contributions that it wouldn't have received otherwise if there's no viable alternative.

Or to give another example, Rust is dual-licensed under the Apache License, Version 2.0 and MIT. This permissive license made it really easy for lots of people (including myself) to contribute to it. If it were released instead using the GPL, it would likely be a shadow of the language it is today, if even still alive at all.

Because that has been my experience in like, almost every employer and customer I have work for since the mid-90's.

Working in places that were based off permissively licensed software, the default position we had was "no upstreaming" and if I wanted to upstream something I need to ask for permission.

when you're contributing to GPL software, you don't need your employer's permission for it to be upstreamed. If they release the code, and don't violate the GPL, then there's nothing the employer can do to stop it from going upstream.

> when you're contributing to GPL software, you don't need your employer's permission for it to be upstreamed.

Yes you do. Except the permission in this case is "permission to use the library" in the first place, which is a much higher bar than permission to upstream changes to a permissively-licensed library because using a GPL library has much farther-reaching implications than merely contributing changes back.

> when you're contributing to GPL software, you don't need your employer's permission for it to be upstreamed

Yes, you do; if you are contributing to it, you need to have exactly that permission. If you are working on a derivative, you need that permission before you can contribute it to anyone else, including upstream.

> If they release the code

Plenty of people work on internal code for their employers, so this is not a given.

> If using GPLv2 means companies stay away from the project instead of building on top of it, that's not a win for anyone.

Like companies staying away from Linux or GCC? It's debatable if these projects would have been as successful using MIT/BSD.

> If using GPLv2 means companies stay away from the project instead of building on top of it, that's not a win for anyone.

Companies having problems with GPL is a problem of companies and not a problem of the license. If the library you want is GPL, then why blame the project and not your company's legal department?

I'm not sure if you're referring to KeyDB or something else as "closed source commercial software", but KeyDB is BSD licensed like Redis: https://github.com/JohnSully/KeyDB/blob/unstable/COPYING

They sell closed source extensions to Redis AFAICT.

https://keydb.dev/modules-redis.html

Facebook just published a blog about moving petabytes per hour. I read it, I find the design interesting. I don't need it. But if I do, it's good to know that it's possible, that someone has solved, to have a reference and to possibly use their tool if it's free. So relax, I for one welcome whatever folks are working on out there. If it's true, we can learn from it and apply it in order domains too.

> Facebook just published a blog about moving petabytes per hour.

For the curious: https://engineering.fb.com/data-infrastructure/scribe/

Edit: HN thread: https://news.ycombinator.com/item?id=21181982

You should post that.

Absolutely astounding to me, petabytes an hour? That's in the region of a meg to several megs per user per hour looking at their monthly active user figures.

It's mostly telemetry data. /s or not /s not sure.

Mmm it's not only "telemetry data". It's that (e.g. Scuba) and other types of logs, and not only Facebook (e.g. Instagram as well).

Basically, everything that needs logging and post-processing by both real-time systems (e.g. Puma) and batch processing (e.g. all of the data that's ingested and sent to the data warehouse) goes through Scribe.

(disclaimer: I work in Scribe)

Scuba: https://research.fb.com/publications/scuba-diving-into-data-...

Puma: https://research.fb.com/publications/realtime-data-processin...

Does that include Instagram and WhatsApp? Or just FB branded stuff like Messenger and the network app...

Not everything that flows through Scribe is tied to an (external) user, though. Tons of internal systems use it as well, notably anything that logs to Scuba (which is pretty much everything at Facebook. Wide-structured system logs are awesome).

>Facebook just published a blog about moving petabytes per hour.

But it's not like that's through one pipe. We don't talk about how many zillions of tons of steel per minute are moved on freeways.

We're using a Redis alternative called SSDB [1] in production because the entire data set doesn't need to fit in RAM. This saves us thousands of dollars per month in server costs. Like you say, latency gains is nominal. The main reason I see alternative databases used is one missing key feature that's custom to a specific use case.

[1]: https://github.com/ideawu/ssdb

Did you test out ARDB [1] when you chose SSDB? I've been considering these for a while since I know I'm going to hit that choice point though that's not happened yet.

[1] https://github.com/yinqiwen/ardb

Looked at it, but setup for SSDB was easier and already was using SSDB in production. Hesitant to change something that's working so well. Do have periodic connection issues with SSDB but so infrequent it's easy to work around client-side. Using the python redis [1] client with Flask.

[1] https://pypi.org/project/redis/

I've personally written a (somewhat hacky but simple) setup for multiplexing high-throughput job queues for a Rails app across multiple Redis instances because our chosen job framework at the time relied a lot on Lua operations on large sets in Redis, and these could easily result in 100% CPU utilization for the Redis process.

We picked the job queue framework long before we started moving tens of thousands of jobs a second through it with a later feature, and probably exceeded its design constraints - in that instance we effectively were using Redis + jobs as a pauseable and throttleable write buffer for MySQL.

With major tooling changes we likely could have come up with something a lot more elegant, I'm not longer with that company, but our plan was always to get rid of the need for that system to hit MySQL at all, which would eliminate a lot of our need to control throughput with our Redis buffer. Of course, doing that would have taken a lot more time and effort, and in our case doing "the simplest possible thing that would work reliably and serve our customers" meant that we probably could have used a faster Redis instead of multiplexing requests and workers across multiple Redis instances on the same hardware.

Sometimes an "improved" version of a tool you're already using can be really useful, if you find yourself in a bind and the alternative is major architectural changes.

The idle CPU might very well be due to redis being single threaded. It might be hitting is limits and still show low CPU utilization.

Also, you didn't mention that, but I've seen this happening often, you should never use redis as a primary data store if your data is important.

You'd expect a 1/n where n is number of cores available utilization in the case of redis being CPU bound.

I guess I don't know how redis deals with the network, but I'd assume that it handles concurrent requests. If not, then I guess that's the case where you'd see less than 1/n cpu utilization and it could still be CPU bound.

Redis handle all request serially, because of that it is much simpler to be correct and doesn't have to worry as much about distributed gotchas.

Because redis handles all requests serially you need to be very careful if the redis cluster is shared between different apps. You want to share the cluster among apps with similar usage patterns.

If the perf isn’t needed we also have Active Replication, subkey expiries, and S3 integration. Multithreading is how we got our start and the reason most people use us though.

True, it's rarely just raw performance. KeyDB has other advantages like multi-threading and disk-based persistence (instead of being limited by RAM) that makes it better at utilizing your server resources and handling larger scales.

But isn't the point of Redis is to be used as a caching layer instead of persistent layer

But what if you could have the speed of RAM-based storage, and still have persistence? Sounds pretty appealing to me, personally.

You already get this with either Redis or Keydb. The problem is that to avoid the disk being the bottleneck you must write to RAM, then later to the disk (you can't wait for confirmation that it was written) which means that even though a key may successfully be written to the DB, it hasn't been confirmed to persist.

Basically redis can't necessarily guarantee data safety - which is perfectly fine for it's normal use case as a caching layer and data you don't necessarily care if you lose (for example rate limiting relies on tracking the number of requests/s within say an hour - if you lose the data you don't really care). User sessions too, worst case people have to log in again, meh whatever.

The major bottleneck for most applications using Redis is the RAM capacity, not the write performance. Disk persistence can be more than fast enough as proven by dozens of key/value stores out there.

Sounds like an embedded key value store, ie. RocksDB. I've used it with great success in constrained environments and also when pushing cache to the edge for stable, low-latency, high volume APIs.

You can use redis as a write-back cache that serves spikes well and streams the state to disk (or an RDBMS) in a rate-limited manner in the background.

Redis is a data structure server rather than a simple key/value store. It's very useful in lots of scenarios, even if the persistence means it doesn't have ACID semantics.

It would be great to remove limitations like RAM-only capacity in exchange for a slight performance hit, and while also gaining better core utilization. We used ScyllaDB (a very fast cassandra clone) in the past for the cpu/disk scalability but always felt Redis offered better APIs. Now it's a real option.

> ...it's rarely just raw performance. KeyDB has other advantages like multi-threading...

What are the advantages of "multi-threading" other than for performance?

Taking advantage of all your cores. Redis is awkward at high capacity since most servers scale CPU with RAM and you'll end up with most cores doing nothing. You can manually run multiple instances on the same server but now they're separate databases with operations and sharding overhead.

Multithreading IO also reduces the latency hit from disk-persistence and provides more concurrency and throughput, which is a great tradeoff when you don't really need sub-millisecond performance but want the same API across a larger dataset limited by disk instead of RAM space.

That's just performance spelled differently.

Ok fair. Disk being primary storage is still a ops and scalability benefit.

> Taking advantage of all your cores.

That means nothing at all, if performance is not your concern.

>> What are the advantages of "multi-threading" other than for performance?

> Taking advantage of all your cores.

Uh, is there any reason you'd want to "take advantage of all your cores" other than performance?

The question/proposal/dialog we started from, don't forget, was:

1. Redis is already as performant as I want, it's not even close to being a bottleneck, so I have no need for multi-threading, and this is a _very very common_ case, as redis is performant enough for a lot.

2. There are other advantages of multi-threading than performance. (Ie, other reasons you'd want it despite 1).

You seem to just be going around in circles. If 1, why would you care about "taking advantage of all your cores"?

Sure, I concede multithreading is for performance. Still, the second point of having Redis structures available over a dataset that isn't limited by RAM is a big advantage.

If redis isn't multi-threaded the more cores and threads you have, the less CPU usage the machine would show, even if Redis is being stressed.

Except if you monitor per CPU usage, as you should.

Except that you can't, because most operating systems will not use the same CPU core consistently for a single thread. This is done for a variety of reasons, including spreading thermal loads to ensure that turbo boost can consistently kick in.

You're right but still, I don't think it is difficult to realize that a particular process is only using one core. If you think it is, you should probably consider another application to monitor system usage.

If you use cloudwatch, the “should” part is tricky. “Should” you be able to and “should” you do it if metrics were reasonably priced based on their basic required storage / computational footprint? Yes.

However, “should” you in a world where you only have cloudwatch because you don’t want to roll your own monitoring or bring in a third party vendor? Probably not. It’s not a good thing, but it’s a reality. 1/n approximation is your go to here.

For some apps, where you can easily scale app servers horizontally and be stuck in a single redis (because redis cluster mode is not well supported by drivers, or because it can't access some atomic LUA operations) you can get redis to be your number 1 bottleneck. Specially because you can throw more cores at the PostgreSQL so Redis quickly becomes the only thing that doesn't scale.

Not only that, assuming we're not trying to hyperoptimize before code ever hits production, typically as a dev I look for performance gains when a bottleneck/slowdown seems to occur. Redis is a possibility I guess, but it's also not in the top 20 places I'm going to look first. It's plenty fast for nearly every use case I've come across and while I'm sure someone out there needs something faster, I agree - I can't imagine there's broad market appeal for it.

Happens more often than you think with larger tech companies. Especially if one has hot keys.

However yeah, 99% of all deployments could get by with SQLite/MySQL/Postgres performance amounts.

The ability to scale vertically instead of sharding is a very nice sysadmin feature that although isn't solving "must faster than redis" is related to it due to multi-threading.

Reply on news.ycombinator.com