Hacker News Reader

Comment by amelius | Hacker News Reader

I don't like how the article tries to push statistics on the reader. If a CS paper compares a pair of averages, then that gives certain information. If statistics can add to that, and make the results a little more precise, then that is nice. But by no means is it absolutely necessary. And statistics will not give a conclusive result either.

I think that authors should use statistics when they see fit, and when it does not distract too much from the original subject of the paper.

View on HN

Needless to say, I disagree. It can be straight-up misleading to report means without including a more nuanced view of the distribution. You don't need to use a bunch of fancy statistics, but you do need to consider whether your results could have arisen by random chance. That's not a distraction; it's accurately reporting what you found.

Here's one frightening example of spurious performance results in CS: https://www.cis.upenn.edu/~cis501/papers/producing-wrong-dat...

> It can be straight-up misleading

It is only misleading if the reader doesn't understand statistics. There is, imho, nothing wrong with putting all your focus on the subject matter, and skipping the statistics while being frank about it.

Also, if you need statistics to show that your method is better than other methods, then perhaps your method is not really that much better.

If your analysis involves data, you can't skip the statistics. If there is no analysis of data than go ahead and skip the stats all you want. You need to try and determine the uncertainty in your results, both systematic and statistical.

Any time you compare two averages, you are doing statistics, whether or not you report the result in statistical terms. It's not something optional that you can "add on" to provide extra information. If you don't provide some measure of significance, I'm not going to trust that your result has any chance of being real. At best, you don't really know whether the result is real because you ignored the statistics; at worst, you ran the statistics and know it's not real, and you're hoping I won't notice.

IMO it's better to just plot the two distributions that you want to compare so that people can eyeball the difference (or run the statistical tests themselves should they wish to do so). For one thing not all distributions are Gaussian. And the t-test only answers one specific question (i.e., with one given p-value). Then there's people who misinterpret the result of the t-test. Or people who mess with the p-value till they get what they need.

This thinking is part of the problem. While an individual average may just be "a fact", you cannot meaningfully compare two averages without knowing more than their values.

Pretending you can has lead to a lot of muddled thinking.

Assuming distributions that don't correspond to reality can also lead to a lot of muddled thinking.

That's true, but though related, a separate issue.

For example. I compute a arithmetic mean of two data sets A,B yielding means a, b.

I can tell you what the difference |a-b| is without any other information, but I simply can't tell you if it is significant or not.

This has nothing to do with distribution. Given the variances, I can tell you something about the significance (at least in some senses). But without knowing the distribution I can't tell you at all how to interpret the variance.

The point is, as soon as you compute that mean, you are doing statistics. If you do it carefully, you will be able to define what the numbers mean, and what they do not. The fact that many people don't do it well does not change this.

There is no possibility of improving the situation by ignoring how the numbers were arrived at and what that actually means. Sometimes the best thing to come out of it is that you are simply calculating the wrong thing for what you want to learn.

You can do quite a bit without making strong distributional assumptions. There are nonparametric tests for all sorts of things (and the basic ones are covered in good first intro to stats books too).

Eh. Let's see how this goes.

Profession A has a mean salary 20% higher than that of Profession B.

Yet people who are in profession A are much more likely to be in poverty than in profession B.

Yet almost any time someone compares two means, they never seem to come to this conclusion - or even consider it a possibility.

Comparing two means without other details is rarely illuminating, and often leads to wrong conclusions (which are worse than no conclusions with no data).

Reply on news.ycombinator.com