Hacker News Reader

Comment by jcranmer | Hacker News Reader

> I think one of the best things that can be done is to separate data gathering from analysis.

That is a stupendously bad idea. The central theme of modern scientific theory is hypothesis testing: you formulate a hypothesis, and then run experiments to see if the hypothesis holds up. If you run hypotheses on pre-canned data, what you very quickly end up with is p-hacking: running several slightly different hypotheses until one clears the significance bar. P-hacking is already a problem that occurs in practice, particularly in the social sciences (it might not be far off the mark to say that most of the results, particularly the most hyped ones, are the result of such p-hacking), and your proposal would make it more readily available and acceptable.

View on HN

I agree. Separating analysis from data gathering is generally a bad idea (especially if anyone with Matlab is doing the "analysis"). Having signed hypothesis and methods before data collection would be good. In that case different labs could do the acquisition (as long as methods for acquisition and analysis are established before hand). Even better to make the acquisition part double blind. There is a strong infrastructure to do this with medicine, but a lot of scientific study require technical expertise that is limited. In medicine there are hundreds (thousands?) of labs or more that can do a double blind study. In a narrow scientific field there may be only a few experts in the world.

It is a difficult problem, but I do think there are better ways to organize scientific study.

To give a counter example: Kepler derived his laws from raw data provided by Tycho Brahe.

I have difficulties to understand how p-hacking would be a problem in physics, for example. There may be fields open to it (particle physics?), but others certainly not, such as astronomy (afaik).

You are of course correct. Unfortunately, the OPs suggestion that 'big data' is changing science has become the common belief in many fields, particularly the most prestigious. Several groups now operate under the protocol: collect big datasets first; spin a scientific story second. These fields often focus on and reward ever increasingly fancy (and expensive) methods to collect more data at 'better' resolution, with respect to some metric.

The hypothesis is often a forgotten tool.

Reply on news.ycombinator.com