Who's Afraid of Peer Review?
sciencemag.org
sciencemag.org
Edit: This looks remarkably like an attempt by Science to smear their open-access competitors, given that it didn't even bother to test whether non-open-access journals have the same flaws (and in fact we know they do, including Science!), yet it's being presented as evidence that open access journals specifically are flawed. I'd hope this article would never survive peer review.
That may be. I wish this had been done differently, and I wish "open access" were not the term used. It may be true that the author is trying to paint with an overly broad brush, throwing some FUD at the movement to make scientific papers easier to get at; I'm not sure.
But it is clear to me that this article is describing a real problem. The problem is the recent proliferation of new journals with very low standards.
> ... and in fact we know they do, including Science! ....
I don't agree. Problems with the peer review system have been demonstrated. It is inconsistent, sometimes lazily done, and subject to corruption in some fields.
But if someone were to claim that you could take 300 journals of the caliber of Science, send them all a worthless paper, and get acceptances from half, then I would call that ridiculous. The occasional acceptance of a poor paper means that the review system needs work. The acceptance of poor papers as a matter of course indicates a lack of standards. There is a difference.
It is true that it can be difficult for someone not in the relevant field to tell the difference between different kinds of journals. I know the difference, since I've heard of the major journals in my field, I know many of the editors, etc. But a science journalist might have more trouble with it.
For fields like medicine, this can matter a lot. It's the same problem we see with SEO and the like: how, in the modern world, do we form a robust measure of reputation and reliability?
Like you, I get (about 1/week) blind offers to submit articles to this or that journal, and even (at a lesser frequency) offers to be an editor (unpaid!). It stinks to high heaven [1]
Getting back to Science, their individual subscription rates are reasonable (~$150/year = 52 issues), their public health articles are available immediately after publication, and their other articles are available after 1 year. They are run by a non-profit institution (AAAS), so they are less motivated to jack up their fees.
I was not able to find institutional fees for Science.
Incidentally, the situation with Nature is different -- they are for-profit (http://en.wikipedia.org/wiki/Nature_Publishing_Group) and it seems to show in their rates: http://today.ucla.edu/portal/ut/uc-libraries-faculty-protest...
[1] Here are the most recent 5: "International Transaction of Electrical and Computer Engineers System" (sounds like a joke), "Entropy", "The Open Software Engineering Journal", "Journal of Robotics", "The Open Electrical & Electronic Engineering Journal".
The internet has many opportunities for academic publishing, but many dangers as well. Academic publishing has relied on the barrier to entry in publishing to filter out incredibly bad journals, or make it clear that they are scams that simply charge high publishing fees; now anyone can create a journal, with low publishing fees, that may be quite legitimate or may be a scam.
And I say this having had a terrible time getting a decent review in recent conferences. But my journal review process has been largely positive, helping to improve my paper significantly, even if they miss some things. Certainly, when I receive a paper to review, I and my colleagues give it our best shot, but we largely work only for the established upper-echelon in our field.
while often true, science and scientific journals do not serve solely the scientific community, they serve humanity, which includes the non-professional public. while you may be an honest person and would never knowingly abuse wrong information to harm others, gain, or advance a broken agenda, plenty of people do. the "vaccines lead to autism" people, for example, do that.
the history of science shows that careful, accurate and correctly performed experiments and publications are what give humanity progress. their absence, history has again shown, causes humanity to retreat, sometimes with disastrous consequences.
this is not an idle, "i sure wish more 'scientists' could publish" debate, at some level the very question of human progress is at stake.
Without a comparison to conventional publishing, these findings cannot be extended as far as the author does in the conclusions. Such an assertion, without support, is not justified or supported.
Recommendation: Resubmit after Major Revision.
* There are bogus open access journals which exist solely to make money from author fees.
* There are bogus conferences that follow a similar business strategy.
* There are lower-quality journals and conferences that do not attract many top-quality submissions, so tend to have a lower barrier for acceptance: you might get work that's rigorous and unexciting, work that's exciting but preliminary or rough around the edges, or work that's neither rigorous nor interesting.
* Peer reviews vary widely in quality, from being thorough to overly critical and nit-picking to being cursory. This is extremely frustrating to many people, but is hardly an undiscussed issue. One particular problem is that more junior reviewers are often reluctant to criticize a paper unless they're very confident in their understanding, either due to it being slightly outside of their core expertise, or due to convoluted presentation: it's easier to assume it's correct than to go out on a limb criticizing something.
I would say that being published in a peer reviewed article is just the start of a gauntlet that any serious scientific claim has to run, regardless of the quality of peer review. Plenty of important work has started out as being extremely controversial: even once it got published people didn't necessarily think it was right. The consensus does continue to evolve.
I've never met a scientist who will believe something just because it's in a peer reviewed article.
Editors have practice spotting cranks but trap papers don't have that smell about them since they are written by smart but naughty people trying to trick reviewers.
Edit: There _are_ shitty journals that no one reads. Everyone in the business knows which ones these are. There is a problem explaining this to lay people, since the shitty journal web site doesn't have a sticker on it saying "zero credibility - please ignore these articles". This is not really a problem for researchers.
My real point was that submitting blatantly bad articles serves a purpose, which is to test whether the current process of reviewing articles is effective at weeding out bad articles. We should not blindly assume that journals, even top-tier journals, are only publishing good articles.
Now, while I understand how journals can assume good faith, they should never assume competence. They are supposed to make sure the paper is good, dammit! If they let that one slip through, they are bound to let genuine fatal mistakes slip through.
No matter what, papers should go through actual peer review before publication. This study, if conducted the way I have read it, is solid evidence that many journals do not do that. They broke trust in the first place.
Anyone whos ever tried to publish. I used to love science before I actually became an academic and realised how dark and corrupt it could be behind the scenes, I suppose the same stands for every field though.
That is not my understanding/observation, some fields seem to be considerably worse then others, while some seem to have relatively little.