Hacker News Reader

Comment by DavidSJ | Hacker News Reader

It means you have to develop your experimental method prior to seeing the data, so you can't try a dozen different ways of slicing that data up until you get the result you want.

View on HN

...and now knowing this, I wish it were the approach for _all_ scientific research.

Yeah. This is what's absolutely missing in many scientific fields, and seems to be the reason for so many results turning out to be utter bullshit (hello, psychology). There's a strong push coming from research community to make this approach, which is known as 'study pre-registration', mandatory.

Presumably pre-registration could also help with positive results bias.

Assuming the registration was made public, one could mandate that something had to be produced from the study, if only a statement that nothing was found.

Wouldn't solve the problem but would presumably aid meta-studies?

It's not really feasible for most research (although there are plenty of needed improvements along these lines). Big physics projects are unusual in that they are enormous undertakings of 100s or 1000s of PhDs, and few or no other experiments will take data that can cross-check the results. (For instance, no other machine will produce the conditions at the LHC for several decades at least, which is why they go to the fantastic expense of building two completely separate general purpose detectors.) Devoting ~5 full-time PhDs for the sake of super-duper methodological rigor is doable for LIGO, but not for smaller experiments.

Well, keep in mind that if hundreds of scientists take their chance on the same data using different methods, we basically have the same problem. One of them will be the lucky one that took the approach that shows the favorable results, whether the results are truly there or not.

Reply on news.ycombinator.com