Hacker News Reader

Comment by subroutine | Hacker News Reader

To clarify, instead of "is correct" I should have said "if our math based theory is being treated as the null". But yes, I completely agree with your statement.

As for you not understanding the first bit, that might be a personal limitation, or some sort of aphasia.

Setting the bar at 7 sigma means we will only reject the null (our math based theory) if the observed experimental data would have only occurred by chance, under the null, with a probability of less than one in ten million.

Hence, setting such a high bar to reject the math based theory means we are relatively biased towards preserving our mathematical preconception about the world rather than accepting (unexpected) physical evidence. If we set the bar at say, three sigma, the bias shifts towards accepting the physical evidence.

The point is, the p alpha criterion is not what makes an experiment rigorous.

View on HN

>Hence, setting such a high bar to reject the math based theory means we are relatively biased towards preserving our mathematical preconception

You mean the only causal and verifiable and by far the most useful view about the world?

Do you think the microchip in your Phone that you post from was built off intuition and gut instinct?

>rather than accepting (unexpected) physical evidence. If we set the bar at say, three sigma, the bias shifts towards accepting the physical evidence.

So the petabytes of experimental data from the LHC don't qualify as physical evidence in your bizarro world. Got it.

This reads like the take that an overly impressed with themself 1st year Psych major would have after taking an experimental methods course.

I am making a single claim, that shouldn't be this controversial. Not sure why you are angrily misrepresenting what I've stated.

Here it is again, more generically: as the p-alpha level for rejecting the null hypothesis decreases, the tendency for Type-2 errors increases.

Do you disagree?

In a world where time and resources are not a factor, an infinite amount of data could be collected to ensure both types of error are minimized. In such a world, alpha can be made arbitrarily small. In worlds that have limited resources to dedicate to experiments, a judgement needs to be made as to whether we are more comfortable with Type-1 or Type-2 errors. If for a given experiment it is decided to set an extremely low alpha (e.g. 7-sigma) it is being implied that we would rather preserve the prevailing theory (null) than reject it in favor of the experimental data.

If a social science experiment was funded at the level of CERN/LHB I have no doubt they would choose a more restrictive alpha, and likewise if LHB had less resources they would likely drop their alpha accordingly based on some power analysis. I say this as someone with a phd in a hard science. Obviously it doesn't excuse the field completely for their reproducibility issues. They should make a more concerted effort with their limited funds, perhaps using the model of particle physicists who pool funds to make a fewer number of experiments more rigorous through higher amount of data collection. But this is all aside the small point I originally intended to make..

Reply on news.ycombinator.com