Hacker News Reader

Comment by yuriks | Hacker News Reader

The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client.

So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead look ridiculous, when it'd be very much acceptable if your server did more actual work.

View on HN

I took the article as to be more a case of using Redis as an example of a system which is reaching the limits of performance _given the current state of the kernel it's running on_, rather than a direct comment on the performance of Redis itself.

Also, saying that using Redis is slower than using a local hash table is a truism. There are myriad reasons why using a local, in-memory data structure is not viable: scalability and persistence, for example. It's like saying "I don't need a database, I can store everything in a local variable."

Having worked on an embedded system that dealt primarily with network I/O (10 Gbps) and hash tables (in the 10 GiB range), I can tell you that network I/O (done right!) and parsing account for MAYBE 1/3 of the total latency of an operation involving a hash table lookup. Memory, like all mass storage, is SLOW once you're not working in cache, and hash tables have no locality.

(By "done right!" I mean either batching requests to/from the kernel, or using a zero-copy userspace solution like DPDK. Clearly in this article network I/O was not done right. Round trips through the kernel ALWAYS will kill performance.)

I've no doubt memory is slow, taking up to 100-300 cycles in NUMA systems. But a single threaded server accessing local memory won't may those costs as much. Are you saying that a few random memory accesses are slower than sending and receiving a packet on two machines?

Redis has a great position as a persistent, shareable, data structure server, but replacing in memory hashtables where they work doesn't seem like one of those cases.

Sure... on a 10 Gbps link, you can transfer up to 15 million packets per second (assuming a tiny payload... which is about all you need for a hash lookup). That's about the same order of magnitude as the latency of a single memory lookup that misses the cache. In the right environment (not TCP on a stock Linux kernel!) each packet carries very little CPU overhead.

Of course, you can pipeline memory accesses to some degree, but not as easily as you can aggregate network requests into fewer packets.

I'm certainly not saying the network overhead is free -- it's not! I'm just saying it needn't "eclipse" the hash table lookup itself (as the GP suggested). They're on the same order of magnitude.

Alright but in the case of Redis, this low latency networking like RDMA or whatnot isn't available right? So, today, off-the-shelf, installing Redis on a remote server, you're probably gonna be off by at least an order of magnitude, no?

Also, on the target server, you still have at least one cache miss to retrieve the item from the hashtable (and potentially more for large hashtables).

It would be nice if there was a commonly supported API for doing this kind of stuff that doesn't require exotic hardware and came with a usable abstraction something like TCP. I know off-the-shelf Intel NICs have "Direct NIC Access" which can easily do wireline work from userspace, but afaik that's only for Intel NICs, and the API isn't as smooth as most socket users are used to. We need something like SuperSockets, but written to target NIC hardware directly, I guess.

There's two groups of people who need to squeeze all the work out of every cycle they can get: Embedded programmers (which, despite our massively powerful phone processors, still includes mobile due to power issues), and cloud programmers. Cloud people are totally interested in optimizing everything to within an inch of its life, so it's a valid concern for them that even if they reduce their user space costs to 0 they still have limits put on them by the kernel. And cloud people are willing to put a lot of cleverness and work into their core pathways, and will get surprisingly close to the minimum time necessary, so you might be surprised how much they can get done in fractions of a microsecond in their core workload.

You may not have these problems, in which case in addition to "the concept of redis" baffling you, this will seem absurdly performance-sensitive to you. From a desktop programmer or all but the most complicated websites, that is also a sensible perspective. But the niche in which this discussion makes perfect sense is itself pretty large.

Game developers also belong to that group of people who squeeze work out of every cycle. Less than 18 months ago, two of the most commonly used devices contained: 512MB dedicated RAM (with 10MB of VRAM) (xbox 360) and 256MB RAM and 256MB VRAM (PS3)

alongside very aged processors. The hardware was almost 10 years old in both cases. Even the current gen aren't particularly powerful, coming in at 8GB ram with a 1.75GHz processor, and a GPU comparable to a 3-4 year old PC for the xbox one, and 1.6GHz processors, 8GB ram and a slightly beefier GPU in the case of the PS4.

And HPC guys, and HFT guys, and graphics guys... Cloud schmoud.

The value of Redis is never going to be found with a single server. It's going to be found when you use Redis to synchronize the state of multiple servers.

Redis can actually be very useful on a single server, too, as a fast, robust, convenient, potentially shared datastore.

Performance is usually a secondary concern in these use-cases.

Yep, for data stores (caches, etc), Redis is awesome and super fast, though not as time-tested as memcached.

And all of the Redis set/list operations are super valuable. You just need to be careful once you start relying on Redis at scale for things you take for granted when you start playing around with it... For example: zunionstores on lots of large sorted sets. "O(N)+O(M log(M)) with N being the sum of the sizes of the input sorted sets, and M being the number of elements in the resulting sorted set." When you start off using it, it's awesome and super fast, but before you know it, the blocking, single-threaded architecture will crash and burn if your data scales up. Luckily we have clustering now :)

It makes a great inter-process work queue as well. I don't think you have to stretch your imagination very far to find interesting single server use cases for redis.

There are better, more reliable ways to do this than redis. NSQ for example, in a reactive (or log-based) pattern.

Regardless, your goal should be reducing shared state as much as possible, since synchronizing it among distributed systems is a Hard Problem(TM).

I've used Redis as bounded temporal storage for "log data" -- specifically, on an occasionally crashy Solaris box w/o capabilities to store a days worth of tcpdump data, I would pipe it to Redis w/ timeouts and have a 15-minute sliding window of data -- so when the host bit the dust, I had what I needed, stored safely. That was valuable.

Let's say you have a 8gb hash table that you want shared between 32 app servers. You are only setting or getting a few elements each request.

It doesn't make sense for each app server to have an 8gb hash table in their memory nor can they easily be synchronized.

But if you do it right you don't parse anything, you take a chunk of bytes off the wire and just cast them to a struct, which you know because you defined the protocol in the first place. This is how market data systems do it.

I've never used a system like Redis, but clearly the trade-off is one of performance for scalability.

Reply on news.ycombinator.com