Type I and type II errors | Hacker News Reader

Type I and type II errors

en.wikipedia.org

17 pointsaburan288 comments

In main street interpretation of statistical findings, type II errors are often swept under the rug.

A good illustration of the danger in only telling the type I side of the story is [xkcd/1132](http://xkcd.com/1132/), the frequentist looks hilariously stupid, and we all hail the Bayesian smartness.

Yet a closer look from the frequentist view, the type II error of the test: H_0: the sun has gone nova is Pr[detector reports no|the sun has indeed exploded] = 1 - 1/36 > 95%; in other words the power of the test(1-Pr[type II error]) is extremely low (1/36). Even though the test has a reasonably low level alpha=0.05(type I error), it's still pretty useless as its power is also pathetic.

The comic made the frequentist look idiotic, however, he's just not a good frequentist with a sufficient grasp of type II error. In reality, many people are like the freqentist, easily satisfied with a low p-value without thinking about the fundamental usefulness of their tests (power), becoming victims of type II errors.

EDIT: I made a mistake, see eli_gottlieb's comment below for correction.

Actually, the comic specifies a 1/36 chance that the detector lies, no matter which result is true. So the P(detector reports no | sun has in fact exploded) = 1/36, making it a very strong test. The real problem there is that the p-value just doesn't mean what he thinks it means, and he in fact hasn't factored in any prior for how often the sun actually explodes.

Oops, you are right, I am a bad freqentist. Just had the discussion about the comic with a friend verbally, thought we figured it out. I guess it is really easy to get even the most basic concepts applied wrong.

And this is why I am currently in my office at 8:15pm reading Jayne's probability theory book.

And my favorite: Type III Errors, wherein the very premise for the calculation is flawed. Always be wary of precisely answering the wrong question.

http://www.tandfonline.com/doi/abs/10.1080/01621459.1957.105...

This is one of those fundamental things they should just make mandatory in grade school. I constantly find myself explaining this to people I work with - and even though it's kind of intuitive ("penny wise, pound foolish", "straining gnats to swallow camels"), it helps to have a formal concept like this to hang the concept on.

The terms "false positive" and "false negative" are so much clear.

True, but we get to sound a lot smarter saying stuff like "type I error" and "type II error".

Reply on news.ycombinator.com