Hacker News Reader

That doesn't affect Jabbles's point in any way.

Run enough tests, and you will get statistically significant but bogus results.

Of course, that's the whole point of declaring statistical significance. With 95%, you will have false positives. With 99%, you will have false positives. There are no guarantees.

Many times, when we declare a winner in test and if you keep running it, you may see that eventually it does not perform as good.

But I agree my comment doesn't negate Jabbles' point.

Ah, but do you understand that the chance of a false positive can be much higher than 5% even with your "95% confidence limit"?

Reply on news.ycombinator.com