Averages Can Be Misleading: Try a Percentile (2014)

We've switched to box and whisker plots for most of our statistical reports, they give you a good idea for various important percentiles and adding average indicators is fairly simple if desired. Even for things like latency this can be quite useful.

> IMHO plotting the distribution should be the first step before trying to compute its statistics. If you know the shape, you can understand the values - otherwise it's guesswork.

My manager's dictum is histogram and time series to start, which falls under the same auspices.

Kibana can do visualizations of distributions with elastic data. I agree it is the best way to get a clear picture of latency and other metrics within a period of time.

I think percentiles over time, given consistent sample sizes per bucket, can be good for understanding change over time.

In general I think of histograms for these sort of metrics providing higher fidelity of value, where as stats over time can provide higher fidelity of time (for roughly the same storage).

Animating the distribution over time can be pretty neat, but not common feature in my space (but can be done with a little R code).

I imagine 3 axis (3d) histograms over time can make some neat visualizations but have never experimented with that, and would probably want to slice or rotate the plot. More common are heatmaps where the darkness of the color is your third axis. Kind of like viewing the 3d plot of histograms over time from above where the cubes are partly transparent and get darker the deeper they are.

Not a bad idea, but in my line of work, tools like statsite calculate the summary stats and throw the data away because aint nobody got storage for that. From this viewpoint, it's still important to teach engineers the mean is heavily influenced by outliers. If they're looking to know what the 'average' is, a median is often better, and if they want to know how broad the distribution is, p95/p99 is a good slice into the long tail.

Agreed, this is actually demonstrated by Anscombe's quartet, a set of "four datasets that have nearly identical simple descriptive statistics, yet appear very different when graphed" (https://en.wikipedia.org/wiki/Anscombe%27s_quartet)

I'm not sure that's really relevant here, since this is monitoring over time. When you design this system sure you could look at the graphs of data, and you'd still end up with the same impression. The point is that over time you can't collect and analyse all the data always.

The important thing here is understand the underlying system and identify what it's modes of failure are. If the likely mode of failure was just to fail to serve some requests at all then the new approach in this article would be entirely useless since the latencies would be the wrong metric completely.

In this case they identified one mode of failure is for a small % of requests to deteriorate almost exponentially and so watching the 99th percentile is useful - this is very common in networking. That's great for this system but the overall message is to find the metric that actually reflects the ways your system will fail- and you need to do that in a practical way that may not necessarily involve recording and storing all the data.

I felt a major point of his talk was you should also look at your max latency.

Specifically Chapter 4, under "Aggregation" https://landing.google.com/sre/sre-book/chapters/service-lev...

thx.. good summary: http://greatesthitsblog.com/the-black-swan-nassim-nicholas-t...

Having multiple percentiles is key, but I can't think of a time when I've ever found average to be useful.

Percentages without volume are still useless though. 99% percentile of 1 billion requests could still mean it's impacting a huge number of users.

moral: No one number is a panacea

Percentiles can be misleading if the data follows a Poisson distribution, especially if the lambda coefficient is closer to 1. Latencies I would imagine, would typically be Poisson due to arrival time.

> the 99% at 867 ms latency makes you have a moment of panic, but when you see that 95% is 60 ms, then you really realize how few of your visitors are experiencing the slow response.

And then you realize that visitors are hitting your application with hundreds of requests per page load and the 99th percentile suddenly becomes your average.

And then you realize that you didn't plot the windowed maximum and have some crazy hangs every now and then that block entire page loads for a whole minute.

> but that Percentiles can be misleading as well.

I'm not sure I agree. If they're computed wrong, sure, but this is what your system is actually doing. And honestly, the tail has a way of dictating your system's performance as a whole.

> the 99% at 867 ms latency makes you have a moment of panic, but when you see that 95% is 60 ms

It's easy to write off 1 in 100 users, but the reality is a little more dim. If your P(slow request) is normally distributed (it isn't always -- some requests are more expensive, some data is on worse disks, etc.), then you can compute the (extremely rough) probability a user will run into a slow request in a session:

P(slow request for user) = 1 - (0.99)^N

N = number of requests.

For example, lets say a user visits 15 pages in a session with that call in each. They have a ~13.9% chance of running into that 99th percentile. :(

Now if you're fanning out lookups (as one often does), you could easy have 50 lookups for a single request. Now you're at 39.5%! What happens in 1% of requests can become extremely important and essentially dictate your user's experience.

The Tail at Scale [1] talks a lot about this. I'd recommend it as reading.

[1] https://blog.acolyer.org/2015/01/15/the-tail-at-scale/

I know it's in the spirit of the talk, but the histogram at 10:45 and the related discussion about how the latency improved for most users but the average latency increasing meaning a worse experience for other users reminds me of the anecdote a Google engineer had when Youtube started rolling out their HTML5 player, the responsiveness of the page had increased but the average latency graphs went up. This wasn't down to it being a bad update, or some users getting a worse experience - not really anyway, but the switch to the HTML5 player allowed a wider audience to start using Youtube where they wouldn't have been able to do this previously. A change increasing average latency, even on a histogram, doesn't necessarily mean it's a bad change. Look at your data indeed.