Hacker News Reader

Comment by Vinnl | Hacker News Reader

Unfortunately, the main reason these journals get used is not because they are better at selecting reliability, but because they provide authors with the credentials needed to advance their careers...

View on HN

But the reason they are considered qualified to serve as credentialing authorities is, in part, due to their perceived reliability. Of course the significance of published work and its influence on the field is perhaps the primary driver of a journal's reputation for readers, but significant work can only be influential if it is also reliable. Were a journal's readership to realize that its publications could not be trusted, it would lose influence and thus the perception that having work published in it is a meaningful professional achievement.

I would like to see these sorts of analyses done over time, although I'm not sure it's possible (in the very least it would require a tremendous amount of work).

Anecdotally in speaking to people who have been in academic research for awhile, things started changing in the late 90s, plus or minus several years. I've heard a similar story repeatedly and independently about different aspects of the system, but in aggregate there's some sense that there has been change in academic research culture that started around that time.

I wonder if the relative quality signal of different types of journals has changed dramatically over time.

Is it, though? Are researchers even aware when previous research has been retracted? Retracted research often continues to amass new citations (which, of course, need not necessarily be positive).

Although I would certainly not assert that research in the "top" journals is particularly unreliable, it is not unbelievable that, when they tend to publish more extraordinary result, a smaller percentage of that tends to be true. I have not seen any indication that that leads to such journals losing influence, and it certainly doesn't lead to the Impact Factor being lowered - which often influences the "perception that having work published in it is a meaningful professional achievement".

The Retraction Watch website http://retractionwatch.com/ is one good method of highlighting retracted work. For researchers in any given field, keeping current on literature is hard work given the pace of publication but does include seeing retractions.

So, the answer to your question depends on which researchers we mean. Experts in a field have the best chance of noticing relevant retractions in their own field. If you are a researcher in one field who has interests in many other fields, it is probably much less likely that you will be able to follow the retractions in those peripheral fields. That is why Retraction Watch and other mechanisms of highlighting such errors is useful.

So runaway social proof, then? Sounds about right.

I've always perceived the superiority of such journals to be: a) rigorous standards for whether the paper's methodology tests their hypothesis (valid science), b) high impact of the hypothesis on the field (utility).

I don't think the OP disputes this, but it does corroborate the old saw, "To make extraordinary claims, extraordinary evidence is required".

Thus perhaps the current standards for experimentation not only fail to be extraordinary, but fundamentally are inadequate.

I don't think there's much reason to believe the "top" journals to have far more rigorous standards than the bulk of "just good" journals (which I'm sure have rigorous standards as well). It is plausible, though, that the research with potentially most impact gets submitted to the "top" journals first, resulting in the "just good" journals never having the chance to publish them.

Reply on news.ycombinator.com