The Kolmogorov-Smirnov Test | Hacker News Reader

The Kolmogorov-Smirnov Test

daithiocrualaoich.github.io

74 pointssteveklabnik21 comments

This is a very nice review, but in practice I've found the K-S test to be much less useful than it initially appears:

1. Failing to reject the null hypothesis is not the same as accepting the null hypothesis. That is, concluding "these data are from some distribution X" is spurious.

2. There's a 'sweet-spot' for the amount of data. If you have too few samples, it's very easy to fail to reject; and if you have too many, it's very easy to reject (the chart at the bottom of the "Two Sample Test" section illustrates this).

3. The question "are these data from some distribution X?" is usually too strong. It's usually more informative to ask "can these data be modelled with some distribution X?"

Nice summary. I think it's worth noting that all of these criticisms generalize from the K-S test to any kind of Null Hypothesis Significance Test.

Agree with you on all three, but specifically for 1., can you think of pathological pairs of distinct distribution that the test would often fail to reject?

The article says it's poor at detecting differences in the tails and much better at differences in the medians. So that's where I'd start to find problems.

Playing with the tails make all kind of mistakes possible, but that seems like a criticism that would apply to any attempt to identify a distribution based on sample.

>if you have too many, it's very easy to reject

What do you mean by this? You can choose any significance level you like.

Nice review! I tried using the K-S test once for some of the old Matasano crypto challenges, to determine if the letter frequency of the text after running it through some deciphering algorithm was from the same distribution as the letter frequency of a sample of the English language. Couldn't ever get it to work, though... maybe that's an inappropriate use of the test, or maybe my sample (Pride and Prejudice, IIRC) was unrepresentative. In the end, simply computing the distance of the two letter frequency vectors (sum of squares) worked.

I did it all in ruby at the time, but it looks like rust may have some stats libraries now? Should give it a whirl again this time using rust, or maybe do tpatcek's new Stockfighter game instead.

The K-S test assumes the data comes from a continuous distribution, so count data would mess up the test's false positive rate. You could avoid that by doing a permutation test, or one of the variations of the K-S test designed to account for data with ties.

What a wonderful article. I've learnt several useful things just by skimming it, even if I never end up using Kolmogorov-Smirnov in anger.

I have (almost exclusively in the finance space). I found it rejecting too many null hypotheses to work in practice (e.g. choosing a distribution that you could generate reasonable simulations with).

For choosing distributions, we ended up plotting observations versus various simulated realities and going with the "eye test". Now, of course, there will be those out that that say this is not rigorous enough, but for most phenomena in the social sciences its very hard to find a distribution that fits reality per the K-S test. So you're left with either bootstrapping (which has its own flaws) a distribution or choosing one that's good enough (e.g. passing the eye test).

That sounds more like a job for the KL-divergence.

Why anger?

Used in anger is an expression that roughly means 'used to achieve something.'

I think it comes from 'shots fired in practice' vs. 'shots fired in anger' (i.e. in combat, for the purpose of killing).

To "use something in anger" is a phrase, meaning to use something "for real" - in production, etc. rather than just to try it out.

the KS test is quite sensitive.

if you are specifically testing against the normal dist the jarque bera test might be better, although also rather sensitive (prone to false negatives).

for two samples if you have enough data to bin, the chisq test is also available to you.

This is quite nice! But does anyone else knows more useful websites such as this with down-to-earth explanations of statistical tools?

That the K-S test works is some cute math by the father of modern probability A. N. Kolmogorov.

But that test and many more are part of non-parametric, that is, distribution-free hypothesis testing. That statistics has long been popular in the social sciences. A major theme in such statistics is permutations. Another major theme, and more recent, is resampling.

I first learned about such tests from a book that was sitting around the office, Sidney Siegel, Nonparametric Statistics for the Behavioral Sciences.

These tests are all one dimensional. Once I published a paper on a distribution-free test that is multi-dimensional -- my paper may remain the only such.

These days, see also the work of B. Efron and P. Diaconis.

More can be done.

I know of one book: Motulsky's Intuitive Biostatistics, very little formulas, much focus on upsides and downsides of various stats approaches, opinions etc.

https://speakerdeck.com/jakevdp/statistics-for-hackers is great!

HN discussion: https://news.ycombinator.com/item?id=10244950

I've used the KS test (and several other nonparametric stats) in research a bit. It's a cool thing to learn about, though it's worth noting that a lot of applied research can make do with parametric approaches. Thank you, Central Limit Theorem.

I've heard Bayesian data analysis is the most powerful compared to all statistical tests/techniques. Can anyone clarify/expand?

Reply on news.ycombinator.com