Hacker News Reader

Comment by onurcel | Hacker News Reader

I like the idea but I don't really get why they accept "statistically insignificant" results. What I expect from a scientific paper is to prove (even emprically) something. For example, if a paper claims something like "we show that using method X instead of Y doesn't improve the results", it won't get published on most journals... except this one, which is awesome, but the paper still has to prove that claim.

View on HN

When you open the field to reward research itself rather than p-hacking, everyone benefits.

It's like a flea market. Bargain bin discounts, junk to many, valuable to some, but it shifts the economics of the whole system and lets specialty niche researchers not have to reinvent the wheel.

Also, knowing that a method or study was statistically insignificant is statistically significant for future study design.

Thanks!

If the only goal is p 0.05 and there's many groups around the world trying the same experiment, we should hope that they all publish their non-results otherwise by chance someone is going to have a significant result just by luck.

Doesn't "statistically insignificant" mean papers which conclude something like "we find no correlation between X and Y"?

I think information like that is perhaps useful, at least to avoid wasting time repeating the same research, but publishing that does not prove that there's no correlation between X and Y, only that the things tested didn't show one.

No! It means the data is too noisy to make any conclusion.

not necessarily. it could. but if x and y are truly uncorrelated variables that are observed with unlimited precision, then you still would not reject the null that they are uncorelated

"Truly uncorrelated" is a problematic concept. Most of the time, any two things within each other's light cones, have at least tiny direct or indirect effects on each other. These effects are usually not that tiny in experimental settings because of imperfect experimental tools and procedures.

Like the speed of sound and the height of the table where the experiment is.

Note that the pressure of the air changes with height. For a few inches the change is negligible and very difficult (impossible?) to measure, but the effect must be out there.

I had once a problem with the temperature of the room, you usually ignore it, but it has a bigger effect like a 5% variation.

You can't ever prove that switching to X doesn't change the result. All you can really hope for is putting smaller limits on the amount by which something could be improved.

For one dumb example: if you wanted to see if rats can fly, you might observe 100 rats and see that none of them fly away. But this doesn't prove anything: in reality even if 3% of rats could fly you'd get this result about 5% of the time. You've really shown with 95% confidence is that less than 3% of rats fly.

If you only publish the positive results, you miss the context of the unsurprising results. For example, in https://xkcd.com/882/ , if the results from all the experiments in the comic were published, the public would know that the "Green Jellybeans cause acne" result is probably a fluke.

It should stop other groups wasting their time on the same research, no?

Reply on news.ycombinator.com