Hacker News Reader

Comment by voxl | Hacker News Reader

A basic course in statistics will inform you of why a 99.99% accurate test should be looked at with skepticism when diagnosing a rare disease. Yet we see the fancy 9s and think somehow this many 9s is enough.

View on HN

Sure, but that’s not what we are talking about

Diagnosis of rare diseases is a subset of "true" statements, no?

he's referring to frequency. a 2.9T parameter processing a 1M token data will inevitably produce wrong "false" tokens, guaranteed.

So even if you have a 99.99% pure date set of the "true" variety, the errors will aggregate to some very less than 99.99% characteristics.

Realm of big numbers problem.

Really? Laws of probability works against your argument.

Such a basic course in statistics needs to be scrutinized.

Every breath you take, there is a 99.99% chance that everything is normal, and a 0.01% chance that you breathe mild acid which horribly burns and causes a massive coughing fit.

It probably don’t cause long term damage unless that breath happened to be more important than normal, like while driving right as a child runs into the road.

Would you act differently knowing that you had one of these occasional acid breaths? Even if it only happened once per day on average (0.001%)

And your point is? Laws of probability doesn't discriminate.

There is always a probability a planet killer GRB hits us and we all die. Should we take precautions in our everyday life. Or should we wear helmet while walking on the road pray a coconut not falling on our head is just 99.999%.

You are correct, but this is a made up example, not a practical matter. You’re falling into the same trap as the author.

You must have come to rescue him :) from the "incorrect" trap.

If 1 in 10,000 people have a disease, then a "test" which always reports the patient doesn't have the disease will be correct 99.99% of the time. "99.99% accuracy" should be "looked at with skepticism" in that it doesn't tell you what you need to know to understand the quality of a a test for a rare disease (a classifier under conditions of severe class imbalance); at a minimum, you would want to understand it's false positive and false negative rate, not (just) it's overall error rate.

See example "A": https://en.wikipedia.org/wiki/Base_rate_fallacy

If only stating the obvious was a fallacy :)

You appear not to have understood probability theory my friend. You will never get 100% in this universe for anything. What if "its a simulation" or "a dream" arguments ensures you never acheive 100%.

Bayes probability theory will be a good start for you.

https://en.wikipedia.org/wiki/Bayes%27_theorem

Well, I do agree that all measurements contain error, but the point wasn't that the error rate would be greater than 0% but that a single headline summary of error can't always distinguish between good and bad tests.

Reply on news.ycombinator.com