Comment by yk | Hacker News Reader

Comment by yk | Hacker News Reader

Science is not broken, but the scientific process is severely misrepresented. I can only really speak about physics, but at least there and I believe also in other fields, people do experiments, observations etc. and then write papers on whatever they did. The papers are intended for a highly specialized audience and contain a lot of detail besides some claim and a p value and there are usually quite a few closely related papers. Actually I sat in more than one conference session where every single talk had the title "The X-ray spectra of NGC XXXX." Then someone will write a review article on the field. And that person is highly specialized, has probably read hundreds of papers in the field over the last decade(s) and has hands on experience. That guy knows that if there is no mention of X in the methodology section, then the authors did not deal with X, which may or may not be justifiable. After that someone writes a textbook, that guy will read several review papers, and he has probably enough experience to know that a sentence like "compare the contradictory results of Y" is a indication that he should look into the actual papers and perhaps use very cautionary language or omit the topic entirely. At that point, when it appears in textbooks, a result is settled science, not when the first indications appear in a paper.

On the other hand, science writes want a headline "Something baffles scientists." Well, the scientists are probably baffled because it contradicts their intuition, which is a strong indicator that something is not really working. This something may be a honest mistake by the authors, a poorly understood experimental effect, outright fraud or it may be something worth reporting. The entire process of science writing is geared towards the papers which are most likely to be wrong.

I agree with what your saying. However, I feel publish or perish has infected many fields resulting in a lot of crap instead of more focused high quality work. Few people can risk spending 10 years on something and have nothing to show for it.

It's a negative feedback loop where quality is hard for people outside the field to judge. So, it's just race to see who can lower quality more while still being published.

> Few people can risk spending 10 years on something and have nothing to show for it.

This is a common sentiment, and fortunately for scientists I don't think it's a very accurate description of the real problems we face. I really don't think that there are many good reasons to spend 10 years on one thing when there is a substantial risk of failure. Scientists spend 10 years on problems all the time! But it's either part time, or has measurable metrics for progress along the way that can be published. I mean, you try to go to the moon before Pluto, right? Science (as an institution) is really a lot more like Tetris than people think, except some of the blocks (publications) get moved or destroyed later. But if the blocks are smaller, they're much easier to fit together and build upon, and fault tolerant and so forth, so if they blow up it's less of a big deal. Smaller, more well-defined and robust studies are generally better.

To change analogies to paths instead of blocks, which is a better description of the decision making a scientist goes through (rather than building the edifice of human knowledge), it's a lot more like finding your way through a new city than choosing two paths in a snowstorm that will not encounter anything for 10 years' walk. At every fork, you have to decide where to go and there will be some reason you have made that choice. After some smaller period, you can if you choose report where you've gotten and why. Other people care! They will be happy to read about it, as long as your logic is pretty good, especially if you make nice observations along the way. You can publish this, and probably no one will steal it from you. Maybe someone will... but if you do it right, do it incrementally and let people know what you're up to, you'll at least have your batshit ideas published as unreviewed abstracts and folks will know who made the first progress.

Let's say you want to study long term impacts of taking Viagra which is clearly going to take some time. Sure, you could look at old data, but that always has issues. So, you get funding, start some research, and want to publish something when your done.

Option A: Pick just one thing to look at say cardiovascular health.

Option B: Collect a lot of data, and then look for trends.

Option B has much stronger risks of false positives simply because your looking at more factors, but those false positives seem much more interesting which helps you get published. Worse you can't use any of that data for verification, so you now need to run another 10 year study. Upside, you got published, downside, your results are almost meaningless. Bigger upside, you then get to publish again in 10 years.

Well what about option C use approach B but publish sooner. Now your not only risking false positives, but also extrapolating from more limited data.

Hmm, guess what say nutritionists are going to pick... "Chocolate good/bad for you!"

Well, not precisely.

I don't know how it is outside astronomy (hello parent parent), but inside astronomy it's relatively well known that there are four journals that are likely to be scrutinously peer reviewed and which _will_ reject you and which _will_ force you to do further analyses. It is also extremely well known that some journals advertise "In print in two months or less!" Inside the field (so to the people who will give you the next contract), it's obvious where you publish.

In some cases, it _is_ very valuable to put the extra months of effort in to lay down a rock solid piece. For example, in CAS, you can make an extra 10,000 RMB for publishing in a low impact Indian or Chinese journal. You can make upwards of 100,000 RMB (a full year's salary) for publishing in one of the "big four" journals. And up to 3 years salary for getting a Nature or Science publication. This is money being invested by the Chinese government to intentionally combat the plague of falsified Chinese research. I don't know of any other country or system that does this. It is also remarkably easy to get tenure in China, which allows a lot of long-term type focus. But this is a tiny bubble for China as it attempts to brain drain the rest of the world and is not likely to continue for long.

It's a big problem that the media doesn't understand which are which. For example when everyone was raving about the "faster than light" neutrinos a few years ago the first thing _every single person_ in my building said was "The Italians set their clock wrong didn't they?" Half a joke, but dead serious that no one should even consider this until it showed up in a publication.

Another big problem is that only basing your research on verifying others research is suicide for an academic career sing you're intentionally doing _low impact_ science. The only people willing to fund that are people with agendas, ie. BP, Exxon, Coca Cola etc.

IMHO, the process is misrepresented >and< the process is broken. I think the article does a good job of explaining many (but not all) of the reasons I think it is broken.

Reply on news.ycombinator.com