Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers
schneems.com
schneems.com
...and if you have the distribution, you no longer need the average!
Latency as experienced by the end user is dominated by the fat fail, for both technical reasons and psychological ones.
Technical ones are probably the most convincing: it is very rare to have users submit single requests and then be done. Especially in the cloudified, service-oriented stacks of today, even single requests lead to a cascade of requests inside the system. Whenever you have tens or hundreds of requests for a single user, it starts becoming very likely that they hit that fat tail at some point in their journey.
Given that latency is dominated by "outliers", looking at anything but p99 and beyond is meaningless.
What's worse is this: since most people at best look at p95 or p99, they tend to optimise for "the common case" at the cost of tail latencies! They introduce insane variance in latencies making benchmarks better, but things actually get worse for real users.
Sorry, this is a pet peeve of mine.
This is frankly wrong. Performance comes in multiple flavors. Latency is one of those, and there we know that percentiles really matter (see Andrew Certain's section of this talk: https://www.youtube.com/watch?v=sKRdemSirDM&feature=youtu.be... for Amazon's experience).
But for others, like throughput and scale, you don't need to know the distribution. In fact for throughput, the only thing that really matters is the long-term mean latency. For concurrency, it's that and long-term mean arrival rate. I wrote a blog post about it a while back (http://brooker.co.za/blog/2017/12/28/mean.html).
The core point here is that all summary statistics are misleading. You need to be clear on what you care about, and making absolute statements about the mean isn't a good way to do that.
Edit: This came across a bit more confrontational than I had intended. The OP makes some good points, but I think his point about the mean is overly broad.
I couldn't agree more. A few months ago I gave a talk that tried in part to emphasize this point (https://www.youtube.com/watch?v=EG7Zhd6gLiw). mjb, I hadn't seen your post until just now but I wish I'd known about it earlier.
Another hard-earned lesson on many teams I've worked with is that humans just aren't very good at judging the variance that's intrinsic to many [summary] statistics. Even when your system is operating in what a human would consider a steady-state, summary statistics are naturally going to bounce around a bit over time. The variance is often higher for tail percentiles just because the density of the PDF is lower in that region. When faced with a question like "did the behavior of my system get worse?" in response to an external change (such as a config change, a code deploy, a traffic increase, etc.), it can be difficult to come up with a reliable answer just by eyeballing a squiggly time series line.
https://arxiv.org/abs/2001.06561
Containing a survey of the most popular Latency Aggregation methods used in the industry (Prometheus Histograms, t-digest, HDR-Histogram, DD-sketch/histogram).
Dunning uses Mean Absolute Error in his latest T-digest paper: https://arxiv.org/pdf/1902.04023.pdf
Cohen uses Normalized Root-Mean-Squared Error to evaluate sampling schemes, which are equally capable of estimating latency quantiles: https://dl.acm.org/doi/abs/10.1145/3234338
The problem with Relative Error as a measure of accuracy is that it depends on the location of the distribution. The same size absolute error becomes a large relative error near zero and becomes a small relative error farther up the number line.
Another thing about this study is that only one value for the T-digest quality is tested. Of course, the T-digest quality parameter equates directly with compressed size, so it's unsurprising that T-digest's size is fixed throughout the experiment. I also suspect that the choice of data set matters quite a lot in this study. If your latency values were clustered around a small range, then the algorithms like DDSketch and Circllhist will indeed have relative error less than 5% (as they prove) but T-digest will be significantly more accurate.
Relative error is a practical choice, since it allows to cover an extremely large value range (essentially all floating point numbers) with small size ( O(log(range)) ) and zero-configuration. You can't have that with bounding the absolute error.
Also the relative error is what you are interested in most of the time as a practitioner. (200ms+/-10ms; 1year+/-15days)
DDSketch uses relative error for estimation as well.
> If your latency values were clustered around a small range, [...] T-digest will be significantly more accurate.
That is correct!
One point of this example was to demonstrate that merged t-digest can have unbounded errors. In the t-digest paper it was speculated that merged digest have bounded error, but the proof was just more difficult. As it turns out, if you have heavy merges and an extremely large value range, you can get unbounded errors.
The main problem is that people try to summarize their data too early. What you want is a measure for how good or bad a single datum is and only then can you summarize the end result. And usually averages aren't a bad choice at that stage.
Averages have some particularly nice properties when dealing with dependent variables, sums of variables, and when you want to minimize the distance between your estimate and the actual value. However to take advantage of that your measure actually needs to make sense. For companies the holy grail is if you can directly express how much money you make / lose because of that single datum, but failing that you'll need to find something that's at least somewhat proportional to it.
In general I agree that averages aren't always bad. One additional advantage is that it's often possible to generate robust confidence intervals for averages, but it's often not valid to generate CIs for medians/percentiles without introducing other probably flawed assumptions.
The whole point is that trying to average the latency and only then try to figure out what it means is backwards.
Ridgeline plots (joyplots) are severely underutilized.
Do you know of any examples?
In fact, the only big difference between histograms and percentiles are that they're sort of the same thing on a different axis.
If you just want to know _if_ something changed then maybe one percentile is ok (though actually: not great). But if you are trying to figure out _what_ happened (or is happening!), a histogram is really important. Most of the time there are discrete behaviors/factors that are driving performance: timeouts, cache hits/misses, one overloaded host, a canary. The shape of the histogram will help you see those in a way that a single percentile can't.
But consider production alerting purposes. For example, consider database user-lookup time. Use the 95th percentile, or even the maximum value to compare with an alert threshold. In many (most?) real-world cases system trouble shows up as a few lookup-time outliers. If you only look at averages, or even medians, you miss those outliers.
Histograms also tend to be quite easy to manipulate
And it doesn't tell you when you just made all your requests 1s slower.
I’ve had the experience multiple times where simply shifting a graph to max shows that timeouts/load shedding is t working, then teams get to the point where they are hitting timeouts way more than they thought. Only after working through those issues do you get to actually improving latency.
The upside of simplicity in the number is only overtaken by the downside when you start chasing real time constraints in systems that don’t need it.
In other words: max latency often indicates some degenerate case (assignable causes) that you want to work out of your system to improve its performance.
But what rarely gets mentioned is that there is no one right answer as to which to use.
It's a business decision on a per product basis.
In some cases, it's totally fine if 5% of the customers get an awful response. In some cases, p99 must be sub 5ms or your customers will leave.
This is one of the key areas where engineering and management need to work together -- deciding which percentile is key for which metrics.
Trouble shootong at the session-of-indiv-requests level is tough, so being able to zoom in/out is the power of correlation IDs and observability stacks (vs this kind of monitoring view afaict).
Consider a system where each server typically responds in 10ms but with a 99th-percentile latency of one second. If a user request is handled on just one such server, one user request in 100 will be slow (one second). The figure here outlines how service-level latency in this hypothetical scenario is affected by very modest fractions of latency outliers. If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second (marked “x” in the figure).
"The tail at scale", Jeffrey Dean and Luiz André Barroso: https://dl.acm.org/doi/abs/10.1145/2408776.2408794?download=...