The Latency/Throughput Tradeoff: Why Fast Services Are Slow and Vice Versa

Its an obvious relationship really. In order to get to 100% cpu occupancy you either need the literally perfect spacing of every request or something is waiting somewhere. Its easier to reason about with very big tasks on a single core, 1s of CPU and tasks 0.75s long the best case is 0.75s, but if you put two tasks in there at the same time one is going to take 1.5s and actually getting to 100% CPU usage only allows for 1.3 tasks a second, the difference in latency on this very low queue behavior is a doubling in latency and queues can often get as lot longer than 1 waiting. SSDs are classic device that shows throughput increase with more queuing due to better utilisation but also substantial increases in latency from typically ms to seconds.

Practically speaking the only way to really reduce latency to the minimum of the software is idle resources awaiting work.

This skips over a solution that helps both users, make each request faster. That may seem like an obvious statement but the “solution” they propose is a complicated partitioning approach. This presumes you’ve already made each request as fast as reasonable. In my experience that’s rarely true.

This also doesn’t get into the real challenge, which is the trade offs for tail latency for median (or more typically 99%) latency. Most of the time the things you do to flatten tail latency increase latency lower in the curve.

The author's thesis largely holds for systems that are at least reasonably well implemented. It's easy to do things that hurt both latency and throughput. You can just waste lots of CPU or memory. You can add unnecessary coordination bottlenecks. You can use inefficient algorithms. Therefore, it follows that you can improve both latency and throughput by fixing these implementation details. Quite often, this will be sufficient for both latency- and throughput-sensitive users.

Of particular note: even if you're already operating in a distributed, partitioned etc. environment, improving per-node efficiency often means that you can serve a given workload with fewer nodes and reduce scaling costs. In the extreme case, the less efficient system would require so many nodes even if it scaled linearly that the more-than-linear coordination costs would kill it entirely. In other words, there's always an absolute limit on how "big" your system can grow. To raise that limit you must often address both per-node performance and scaling efficiency.

This is not merely hypothetical. When I was working on distributed filesystems, a competing system that was genuinely more scalable in terms of node count was so ruinously inefficient on each node that they not only couldn't beat us on latency but they basically couldn't beat us on throughput per dollar at any level either. So they resorted to politics instead of engineering, but that's another topic. The key point here is that eliminating waste will typically improve both metrics while adding scalability will only improve one at most and might actually not even raise the ceiling on that.

Once a system is "well implemented" as I've described, the OP's tradeoff becomes relevant. But TBH a lot of systems just never get to that point. They end up being good enough for business purposes even when there's still tons of room for technical improvement. Often, some simple math around utilization levels and controls around batch sizes will suffice.

1. It's a trade-off so real-world implementations are likely to suck at both ends of the spectrum. (eg. Old Ruby servers are neither responsive nor they have large throughput)

2. The trade-off is more about computation in general. In typical "service" scenarios you almost always want to prioritize latency over throughput, while in most "batch processing" scenarios which you want to optimize overall processing time you want to prioritize throughput.

Examples:

1. Erlang server: Great preemptive scheduling thus the theoretical overall computation throughput is relatively lower. But in practice, Erlang servers are extremely fast and responsive. (Prioritize latency over throughput, great for "services")

2. Traditional C/Java blocking (non-async) server spawns threads: Theoretical computation throughput is greater but in reality, it's very easy to DDoS it. (Great for batch processing but suck at "services")

Maybe a knee jerk reaction, but why cannot the low latency users skip the queue, or even throw out a existing job and reschedule it for later - the bulk users does not care, right?

The problem reminds me how we made a lousy internet connection usable across a whole dorm using queues in m0n0wall to prioritize low latency traffic

Wouldn't a priority system enable serving both batch and low-latency requests with one cluster with good utilization? Could be furthermore improved if the batch requests can be interruptible.

More accurately, this is a Latency/Throughput/Cost problem.

The article assumes "max throughput" means "operate the cluster at 100% utilization". But for a parallelized service that's actually assuming they want "max throughput per $". As described, BigTable can have low latency and high throughput if you are willing to overspend on compute.

Look forward to see what the follow-up post has to say.

I've encountered these situations and solved it using common sense that might vary case by case. In one case we had 3 types of database servers: writer, normal-query, slow-query. The last one was added because certain queries would use up significant cpu and i/o.