Hacker News Reader

Comment by RcouF1uZ4gsC | Hacker News Reader

Some observations.

First, I think that the scientific literature peer review process becoming more like Amazon's reviews would be a disaster for science.

Second, although a scientific discovery may take months to publish in a journal, that does not mean that the scientific community is in the dark about it. Many times a finding is presented at various conferences well in advance of it being published.

However, scientific peer review is broken. I think one of the best things that can be done is to separate data gathering from analysis. With machine learning, there will be a great demand for high quality raw data. It would be great if scientists and institutions could achieve recognition for collecting methodologically sound, high quality data. Often times, a high quality data is not made public, but is milked to produce a series of papers by the institution or researcher that are all based on the same private data source.

View on HN

> I think one of the best things that can be done is to separate data gathering from analysis.

That is a stupendously bad idea. The central theme of modern scientific theory is hypothesis testing: you formulate a hypothesis, and then run experiments to see if the hypothesis holds up. If you run hypotheses on pre-canned data, what you very quickly end up with is p-hacking: running several slightly different hypotheses until one clears the significance bar. P-hacking is already a problem that occurs in practice, particularly in the social sciences (it might not be far off the mark to say that most of the results, particularly the most hyped ones, are the result of such p-hacking), and your proposal would make it more readily available and acceptable.

I agree. Separating analysis from data gathering is generally a bad idea (especially if anyone with Matlab is doing the "analysis"). Having signed hypothesis and methods before data collection would be good. In that case different labs could do the acquisition (as long as methods for acquisition and analysis are established before hand). Even better to make the acquisition part double blind. There is a strong infrastructure to do this with medicine, but a lot of scientific study require technical expertise that is limited. In medicine there are hundreds (thousands?) of labs or more that can do a double blind study. In a narrow scientific field there may be only a few experts in the world.

It is a difficult problem, but I do think there are better ways to organize scientific study.

You are of course correct. Unfortunately, the OPs suggestion that 'big data' is changing science has become the common belief in many fields, particularly the most prestigious. Several groups now operate under the protocol: collect big datasets first; spin a scientific story second. These fields often focus on and reward ever increasingly fancy (and expensive) methods to collect more data at 'better' resolution, with respect to some metric.

The hypothesis is often a forgotten tool.

To give a counter example: Kepler derived his laws from raw data provided by Tycho Brahe.

I have difficulties to understand how p-hacking would be a problem in physics, for example. There may be fields open to it (particle physics?), but others certainly not, such as astronomy (afaik).

Your last paragraph is important. Open-access data is a critical piece to good research. Too often its used to gatekeep (As you mention).

Good luck with getting a group of scientist to agree on "high-quality" data collection methods--lol. Not to say we shouldn't standardize things more than they are. To your point about research being presented at conferences--true. But very few people are actually present at these conferences. I actually keep up with lab webpages to stay informed--usually people are good about updating these, but sometimes people dont even have one :/

Labs are good at updating webpages, hah, that wasn't my experience at all and is usually a common to see exceedingly out of date webpages. Other than some high profile groups that probably have money to hire specific webpage staff everyone else has to do it themselves and they often don't keep up.

I suppose my memory is biased (because if i visit a webpage and see it hasn't been updated since the millenium, i wont visit again) in that i only re-visit updated lab webpages.

That said, the new wave of academics (< 35) seem to be friendly to updates, IMO

First, I think that the scientific literature peer review process becoming more like Amazon's reviews would be a disaster for science.

I didn't see where they're looking to have something exactly like Amazon. To be charitable, I'd assume that they would have more controls and reputation-attribution in place to ensure that many concerns would be addressed for a "more open" system.

Many times a finding is presented at various conferences well in advance of it being published.

Do you mean that you have to go physically to a conference to learn about other possibly-important discoveries in your field?

>First, I think that the scientific literature peer review process becoming more like Amazon's reviews would be a disaster for science

You didn't mention any reason for this. While I am skeptical about it too, I also think Amazon's reviews work great for buying a product on Amazon. Also, informal discussions about a paper happen all the time on twitter within the academic community. Giving it a more formal structure is not a big stretch.

Reply on news.ycombinator.com