Hacker News Reader

Comment by RA_Fisher | Hacker News Reader

This is a nicely done article. One of the issues with using a z-score for count data is it assumes the data are normally distributed. This implies values < 0 are possible. However, since counts are lower-bounded at zero, they’re asymmetric unlike the normal. This means that practically this method would produce too many false positives for most applications. Whatever threshold is chosen, is likely to have too narrow an envelope between false negative and false positive to be reliable for count data. However, for normally distributed data, this would work perfectly.

View on HN

> One of the issues with using a z-score for count data is it assumes the data are normally distributed

A z-score is just the number of standard deviations away from the mean a data point lies. It doesn't require any assumption about the distribution of the data.

If you wanted to make some inference, like maybe about the likelihood of observing some z-score given a hypothesis, then you would need to assume a distribution.

But, nothing the author does assumes a distribution, normal or otherwise. It doesn't mention probability at all as far as I can see.

When you decide that a value N standard deviations away from the mean is an anomaly that must be investigated, you implicitly make an assumption about the underlying distribution of the data.

Exactly, it’s a smuggled assumption. A standard deviation is the second moment of a normal distribution. A negative binomial is a much better model for count data: https://statwonk.com/poisson-gamma-negative-binomial.html

The simple approach in the article is no different if you imagine it as a negative binomial. The article does not assume any distribution, it's entirely nonparametric.

That's not the case. Looking for |z|>t implicitly assumes the distribution is symmetric and unimodal for a start. As in, the method would break if those assumptions were violated.

Non-parametric means more than just avoiding talk of distributions.

Good point about symmetry & unimodality. You could easily construct a bimodal example where the method fails entirely.

In my previous post I said "doesn't assume any distribution" I meant it doesn't depend on any particular distribution function.

But in the post above that I was sloppy and said "no assumptions about the distribution at all". Good catch.

That's looking at it backwards---this is a non parametric approach. The author didn't assume a priori that a point with a z score greater than N was anamolous, for some fixed N.

Instead, they determined N by looking at the past data.

(Contrast with the parametric approach of assuming normality and concluding that any point lying >3 sd from the mean is an anamoly due to properties of the gaussian distribution.)

I don’t think this is correct . You can calibrate a threshold for a metric empirically without interpreting the metric probabilistically. Ie that z > N is empirically a good threshold for anomaly detection calls for no necessary further interpretation.

Sure, but for most processes that’ll yield disappointing performance.

Any tool can be used without regard for the consequences, but knowing a tool’s statistical properties yields know-how about the consequences of its application. It’s often a matter of the costs of error / stakes of the most problem at hand. Cheap solutions are often the best for cheap problems.

Good point... maybe first transforming the data to be approximately normal could be a solution? (I'm not a statistician)

My current take on this is that one should use a P99 (or P99.9, etc) median and anything above that is considered an anomaly.

You then are not assuming anything about the distribution.

I'm curious if others think this is a bad/good approach.

Reply on news.ycombinator.com