Statistically rigorous Java performance evaluation | Hacker News Reader

Statistically rigorous Java performance evaluation

blog.acolyer.org

46 points0wl3x18 comments

I prefer reporting the mean and the standard deviation - the paper advocates a confidence interval instead of standard deviation. Typically, I'm more concerned with the spread of obtained performance values than I am with how likely it is that our measured mean is the within some interval. I generally don't think of that spread of obtained values as noise or random errors, but as systematic consequences of using real computing systems. The reason I don't consider that systematic error is that the sources of variation in real computer systems are often the result of things like memory hierarchies and system buffers that will exist in practice. Real systems will have these things, so I want my experiments to have them as well - so long as our benchmark has them in the same way a real production system will have them.

For example, see Table 2 in a recent paper I am a co-author on (page 8 of the pdf, page 73 using the proceedings numbering): http://www.scott-a-s.com/files/debs2017_daba.pdf In this paper, we care about latency, and we report the average latency along with the standard deviation. Here, a tighter standard deviation is more important than confidence that the mean falls within a particular range. And the variation in latencies is caused by both software and hardware realities of the memory hierarchy.

When you want to make a formal scientific claim - that technique A is faster than technique B, for example - how do you support that using a standard deviation as an error?

I sometimes have used standard deviations for errors myself, but I'm not really sure I can defend it.

What I'm saying is that it's not error. I'm reporting a distribution that the values may take on. Ideally I would show all of the data, but that's not possible given both presentation and human cognitive limits, so we characterize that distribution with two dimensions: a mean (the "middle") and the standard deviation (how much it "spreads" from that middle).

This is, I think, a philosophical difference from, say, measuring the charge of an electron. The electron has a charge. The mean of independent measurements is, we hope, very close to that true value. Deviations from that mean are indeed error. But when measuring performance in a computer system, all observable values are valid. We're trying to figure out not what the "true" value is (there isn't one), but what the range of values your performance can take on, and where in that range you're likely to fall. (I almost wrote something about excusing "real" errors such as memory corruption, disk failures and segfaults, but sometimes you want to include that! If you're doing performance analysis of a distributed system that scales to hundreds of thousands of compute nodes, real runs of an application are likely to encounter such things, so your system better be resilient to them, and they will have an impact on real performance.)

For microbenchmarks, I've seen it recommended that you just take the min value. The idea behind it is that any extra time from the min is very likely noise so the min is closest to what you want to measure.

I strongly disagree - I care about how the thing will perform in practice, which includes what people often consider "noise." The minimum value for a metric is just when all of the stars, sun, moon and planets lined up juuuuust right, and is often not representative of real performance. What if, for example, the technique performs great when everything is in the cache, but it also uses the data in such a way that it has horrible cache locality? Your min will be great, but your average, max and standard deviation will be terrible.

Microbenchmarks specifically try to hold the rest of the world constant while measuring a microscopic piece of code, in the realms of countable cpu clock cycles. In that case, noise really is noise. And of course the data is of limited use.

Things like cache locality, branch prediction, prefetch etc. should be flushed out with a good microbenchmark prologue.

Again, disagreed when the point of your technique is better cache and memory hierarchy performance. Much of my own work aims to perform better given the realities of modern systems. Trying to factor such things out removes the thing being measured!

I don't think you can say that microbenchmarks are anywhere realistic. First because the machine workload is different while you run them. Second because the compiling or VM state is also quite different.

A value of microbenchmarks is to establish a stable environment when a baseline can be set, where predictions can be tested. It's a development tool, not a qualification tool. You will have anyway to measure the real performance impact in the complete environment.

Mean and standard deviation seem downright dangerous to use, especially to understand latencies. Both mean and std are very sensitive to extreme values, and these extreme values tend to be frequent enough that shoving them under the 'outlier' carpet does not instill much confidence. Looking at multiple quantiles (if not all of them) would be more sound. But if one is compelled to reduce the data to 2 numbers, one can go with trimmed-means, and L-moments or some other robust estimate of location and scale (EDIT. for example, median and interquartile. Just that trimmed mean is more efficient, both statistically and computationally).

Deficiencies of mean and std are easy to demonstrate. Just draw multiple samples of different sizes from a Pareto distribution (some other long tailed distribution would work too) and observe how drastically the mean and std varies among those samples.

Gaussian distribution based methods, for example ANOVA, had their moment of glory in centuries past (good stats has since moved on). They are still fantastic tools when the assumptions they make are true (this is rather rare) but they break badly when those assumptions are violated a tiny wee bit. One can easily end up making a wrong decision.

At the very least one should verify that the Gaussian assumption holds before believing those statistics. There are many post-Gaussian statistics that one can use, if it turns out that the data is not Gaussian.

In my work, I have never discarded outlier values. We investigate them to make sure we're not doing something boneheaded, but once I'm confident it's legit, I always include it.

I agree that trimmed-means are L-moments are better at characterizing a distribution. But! I had to look them up just now, and I have never seen a computer systems paper which used them. I don't know if most readers would be able to intuit meaning from them. My claim is not that mean and standard deviation is the best way to characterize computer systems performance data, but that for my work, it's better than mean and a confidence interval.

> In my work, I have never discarded outlier values.

This is great to hear, hope everyone follows this.

> I have never seen a computer systems paper which used them. I don't know if most readers would be able to intuit meaning from them.

That's rather unfortunate, is it not ? Current practice is broken. So should one not strive to fix it and introduce the community to better tools, champion better tools. I am aware I am being harsh here but, correctness be damned, lets follow the crowd does not show the community in stellar light. If trimmed mean is too exotic, even median and inter-quantile distance would be fine.

> My claim is not that mean and standard deviation is the best way to characterize computer systems performance data, but that for my work, it's better than mean and a confidence interval.

I have two comments to make (i) Its not the case that mean and std are mostly OK but rather suboptimal. I would be fine with that. The thing is these can be horribly wrong and unrepresentative. Seemingly innocuous looking distributions have infinite variance. Since you use a lot of Gaussian based tools, you would be familiar with a t-test and the t-distribution. The variance of a t-distribution with 2 degrees of freedom is infinite, any finite number one reports for the std would be wrong by an infinite order of magnitude.

> it's better than mean and a confidence interval.

(ii) I am having trouble with this one. Assuming that confidence intervals have been computed using the Gaussian assumption, width of the confidence interval would be a scaled up version of the standard deviation, nothing fundamentally different. Instead of std, you get, say, 3x of std. When they are bad, both are equally bad.

I am not against the use of Gaussian, sometimes it is very reasonable, but if correctness is a goal, then it needs to be demonstrated that it is not a wrong thing to use. There are standard techniques to verify that using Gaussian based stats would be fine.

I would be sympathetic if it was the case that we don't know how to handle departures from Gaussian, or it is very expensive to handle that. Neither of these are true, and haven't been true since the 80s. in fact earlier.

So use the median and the median absolute deviation, or use them in conjunction with mean and standard deviation. Please don't discourage people from reporting distributional statistics.

For latencies I'd want to see percentiles. Mean / standard deviation are pretty much useless for comparison. With percentiles I can get a good idea of the distribution. And with things like HdrHistogram it's easy to measure.

The point of confidence intervals in TFA is to determine whether there are statistically significant differences - but I think they must be making their own assumptions about the underlying distribution to come up with the CIs. So not sure how much I'd trust it.

Percentiles are ideal. But when you want to report the results of 34 experiments in less than a quarter of a column (see Table 2 referenced above), you gotta make some simplifications.

Standard deviation is a good way to measure consistency of a system. I use it in conjunction with percentiles to gauge health of our systems.

More recently:

"Quantifying performance changes with effect size confidence intervals" Tomas Kalibera and Richard Jones Technical Report 4-12, University of Kent, June 2012.

https://www.cs.kent.ac.uk/pubs/2012/3233/

Kalibera, Tomas and Jones, Richard E. (2013) "Rigorous Benchmarking in Reasonable Time"

https://kar.kent.ac.uk/33611/

SPECjvm98 is an outdated measure of both system and JVM performance, the benchmark to look at is SPECjbb2015 which very aggressively taxes JVM subsystems like the GC and the JIT.

Yes, but this paper is old so they did the right thing by using SPECjvm98 at the time the paper was written.

Reply on news.ycombinator.com