The Arsenic DNA paper exposes flaws in peer review
michaeleisen.org
michaeleisen.org
Most scientists I know (physics) don't have this idea. They know publication is just a step. Most grad students learn at some point that just because it's published doesn't mean it's correct.
Publication is a checkpoint, not a finish line, for any theory or experimental result.
Wait, maybe they do understand but they also understand that the general public doesn't understand.
In my experience it's more like: Most scientists ignored it. Others were interested but had too much teaching to pay attention to it. A few people got pissed off and tried to prove it wrong. A few other people got excited and tried to build on it.
I feel all these debates are kind of pointless, because no-one I know treats a published article as gospel. We make up our own minds about the paper.
> If we had, instead, a system where the review process was transparent and persisted for the useful life of a work
That is a very interesting idea. (Reviews are currently confidential and usually anonymous -- only the author and editor sees the reviews, and only the editor knows who wrote them.)
Not only would this reduce blatantly petty reviews, it would also give some recognition to the work that goes into reviewing a paper. It's currently an important but very thankless job. It's considered polite to acknowledge reviewers at the end of the paper (even if they remain anonymous), but it's often not done.
I think you're spot on with this comment. That's exactly what we're building at publons.com. Check it out.
I've always advocated that the reviews behind an accepted paper SHOULD all be public and publicly distributed along with the paper.
I hear so much about how researchers provide papers to journals for free, and review papers for journals for free, and then the journals (are presumably evil to) charge high fees to anyone who wants to read the papers. How much can we pile on the reviewers before they stop doing it?
Note that IMHO is OK to be anonymous while refusing a paper because otherwise a reviewer may be put under lot of pressure.
The bottom line is I, as a powerless grad student, am _very_ comfortable with the level of reviewing competence I have witnessed (and been a part of).
The proposal to deanonymize reviews has a huge flaw: Namely, there's no way I (as a grad student with no political power) will write a critical review of somebody famous if I know I won't remain anonymous.
The risk of blowback is too damn high. (And given the small pool, writing style would probably be sufficient to deanonymize, if my entire trail of reviews were public). There's no way I have the balls to write a critical review of a paper by, say, Robert Morris or Larry Peterson, for some prestigious venue such as SOSP, when I'm asked to write a review. Unless I know for sure that there's absolutely no risk whatsoever of being identified.
Look. Reviewing is thankless and hard. Anonymity exists so that we can be critical and not risk career suicide. Last year at OSDI I received 9 reviews, and they were all exemplarily thoughtful, even the obligatory "strong reject" one of them. The problem of "petty reviews" _has_ a solution: The program committee is self policing, and the reviews are not anonymous within the PC. (For example, at SOSP this year, there are 28 PC members.) You know that most of your reviews come from this gang of 28. You might have one or two externally solicited experts in addition, but that too is not anonymous _to the PC members_. If there's a culture of meanness, the PC members and PC chairs can fix this by internal policing (and they do).
I completely agree that there's a lot value in anonymity when reviewing, and I don't think it should go away. Blowback over disagreeing is very, very real, and I've experienced it myself (though it was from presenting something at a conference that disagreed with part of a previous publication, rather than as a result of a review).
However, I do think public and optionally non-anonymous reviews could be helpful.
(Incidentally, you had _9_ reviewers for a single publication?? Holy cow... In the geosciences, I've never heard of more than 3, and it's usually only 2. That's one thing that would really help in my field. More reviewers results in a broader review, rather than a particular person's pet peeve.)
Incidentally, I have deanonymized several reviews already. When you go to conferences, and some researcher asks the exact same question and has the exact same opinion as reviewer C, chances are... Sometimes the recognition is mutual as they realize what they just revealed (thankfully that has only resulted in humorous chuckles, at least in my life. But I can see how it can go wrong so easily!)
I do know that I have received many cursory reviews where the reviewer did not fully read the paper. I have often wondered if single-blind the other way would be better: reviewer signs their name, but paper authors are anonymous. While this may fix the crappy review problem, it is susceptible to what you fear.
I would hope that those cursory reviewers resulted in feedback from the other PC members that fix the issue (either through improved future reviews, or not being invited back to be on the PC.)
Essentially, anonymous experts review publications in much the manner they do now, with no risk of their identities being revealed. A separate set of review-reviewers then rates the publication-reviewers themselves, scoring them on a standardized quantitative scale (1-10) based on the validity of their claims and the depth of their assessment. In this manner, experts collect an "aggregate reviewer rating" of sorts analogous to an h-index [2], call it an R-index.
This R-index can then serve to select accurate reviewers for future publications. A high R-index could also be included as a positive bullet point on a CV, providing some reward for reviewers to put a high degree of effort into their otherwise unrewarded reviewing work.
A few potential flaws are evident: what is one to do if early reviews are scored poorly, and you are no longer able to provide new reviews to increase your R-index? A grace period before scoring could potentially remedy this, ie) an R-index is not assigned until a reviewer has written >10 reviews.
[1] - http://www.frontiersin.org/Computational_Neuroscience/10.338... [2] - http://en.wikipedia.org/wiki/H-index
But basically existing program committees (in computer science) do serve that meta-moderation purpose (without the empirical R-index metric). The entire PC does get to see the deanonymized reviews of the other program committee members. If someone's just being an unhelpful jerk, they're probably not going to get invited back in the future.
This is not entirely true. I've talked with several people involved in research (math and CS). The ideal is that only the editor knows who wrote the reviews. However many fields do not have that many people who are qualified to review cutting edge results. This means that it is possible to get an idea of who your reviewer is based on how he wrote. Of course, although the people I talked to said they often can figure out who reviewed them, they cann't actually confirm that they are right (and I definitely cannot).
If a paper claims a significant result, it will eventually get tested.
If it claims an insignificant result, it might never get tested. That's ok, little harm done because it is insignificant.
Skepticism is at the very core of science.
Partly to blame are the "non-fiction/science&nature" book category, and the journalist/writer cross-breed who chain-write in newspapers, magazines and churn out books from their articles every year.
At first, the really good ones were a welcome addition to Science literature, for bringing the science to our daily lives. But then they became part of the "where it bleeds, it leads" culture, sans proper journalism and sans scientific thought.
Scientists may make fun of how journal articles are covered in newspapers and blogs, but the negative effects of such coverage are quite real.
I recognize that, in many ways, physics has it easy: there are few sciences that reduce as well as it does and that offer ways to isolate variables like it does (even if it requires multi-billion dollar machines and years to make that happen). But it is still a shining exemplar of gradual evidence building theories to find more evidence, and there are many, many fields that aren't anywhere near that.
Not to mention that research is all about who comes in first; the incentive to validate published work is by and large not there. I recall a Nature article on data reproducibility: only 11% of the TOP, PEER-REVIEWED PAPERS were able to be reproduced[1].
[1] http://www.nature.com/nature/journal/v483/n7391/full/483531a...
The contrast in the public response to the two incidents is fascinating. While Cold-fusion became a career-ender for anybody associated with it and Fleischmann & Pons both suffered pretty much banishment, repeated experiments have shown that there is something funny going on with that particular mix of compounds (Heavy water, Palladium, Nickel etc) - excess heat.
With arsenic life however, follow-up experiments have proven that it was contamination that was a cause of the result. So there is NO ARSENIC based life. Yet the lead authors of pretty much a gigantic false claim -orchestrated shamelessly by NASA- seem to be doing pretty well, dodging the basic questions with that most famous catch-all - more work is needed (hence more money should be sent my way) to prove the (bogus) claims made.
Their mistake wasn't the paper; it was the press conference.
http://coldfusiongelato.com/site/home/
Mmm
There is no excess heat, it's measurement error caused by miscalculating how much hydrogen recombines in the cell.
If you build a calorimeter that recombines all hydrogen you find no excess (but it's really hard to do accurately).
> One of the first to respond to Semmelweis's 1848 communications was James Young Simpson who wrote a stinging letter. Simpson surmised that the English obstetrical literature must be totally unknown in Vienna, or Semmelweis would have known that the English had long regarded childbed fever as contagious and would have employed chlorine washing to protect against it.
And:
> Indeed, initial responses to Semmelweis's findings were that he had said nothing new.
But:
> In fact, Semmelweis was warning against all decaying organic matter, not just against a specific contagion that originated from victims of childbed fever themselves. This misunderstanding, and others like it, occurred partly because Semmelweis's work was known only through secondhand reports written by his colleagues and students. At this crucial stage, Semmelweis himself had published nothing.
Why didn't he publish until later? Who knows. The fact is, he didn't and it hurt his case.
And, if you read further, you see that Semmelweis simply went crazy later on, with people later on proposing causes from Alzheimer's to learned helplessness to PTSD. Not a good thing if you are insulting and argumentative to begin with and are proposing a new theory on top of it.
He was onto something, had data showing the correlation, it was a simple method with no contraindications hindering it's immediate adoption... Still he was faced with incredulity or irrationality. The ones who didn't vilified him, didn't bothered to investigate further either.
You don't need proof to say "this class of reaction might be possible at a different temperature; we're not sure yet"
The default assumption is that there are many many ways to accomplish any particular chemical or nuclear change.
The main point is that someone working at Science wrote a bunch of fake papers and ascribed them to fake African scientists (which bugs me a lot, since it might both assume and create discrimination) and showed that a bunch of open access journals don't do peer review properly but he didn't do the same thing to the conventional journals.
Where's the control group?
Any scientist will tell you over drinks about peers who double-publish, fake data, and force anyone whose work they review to cite their own work. I'm reminded of the scandalous story (in the New Yorker a few years ago, sorry no link) of a famous -- Fields Medallist no less -- mathematician who took over a journal and started publishing his own and disciples' work without review to get the jump on rivals -- and does a science get more assumed credit for perfection than Math?
At the time of writing the posted title here is the same as the article title: "I confess, I wrote the Arsenic DNA paper to expose flaws in peer-review at subscription based journals"
I feel like this shouldn't be too difficult to make.
PLOS ONE is this. It is well respected already. The initial invited reviews are mainly a junk filter to establish baseline methodological quality and NOT to evaluate significance.
From their "About us" page:
"Unlike many journals which attempt to use the peer review process to determine whether or not an article reaches the level of 'importance' required by a given journal, PLOS ONE uses peer review to determine whether a paper is technically sound and worthy of inclusion in the published scientific record. Once the work is published in PLOS ONE, the broader community is then able to discuss and evaluate the significance of the article (through the number of citations it attracts; the downloads it achieves; the media and blog coverage it receives; and the post-publication Notes, Comments and Ratings that it receives on PLOS ONE etc)."
The arXiv focuses on physics, math, computer science, and similar, but it certainly seems to be what you're looking for.
More more notable than the fact that some fly-by-night, pay-to-play "open access" journals accepted a scientifically and ethically dubious paper is the fact that there now exists a large body of established, high-quality open access journals who roundly rejected the paper and pointed out its many flaws.
Meta publication bias strikes again.
So, when you get junk accepted in a humanities journal, its because the field is about bullshitting anyway - when you get junk into a science journal it's a hitch with the review process?
But yes it was and no it didn't. That's always bugged me about the Sokal affair and people holding it up as an example of anything. While the editors did publish the paper, which is a basic problem in and of itself, they claim they wanted big revisions that Sokal refused to make, but then they published it anyway. Since the whole point of that journal at the time was to get wide-ranging viewpoints, I think there is some merit in the editor's claim of a simple betrayal of trust: they assumed Sokal was being straightforward, and the worst that would come of publishing his piece would be a bunch of letters telling him to stick to his day job. In any case, I don't know if we'll ever a definitive story of what really happened, how carefully they vetted the paper, etc.
Regardless, the journal he submitted the paper to was not peer reviewed, so the whole thing demonstrates nothing about the peer review process.
Instead of making the reviews transparent, some journals solicit notes from other researchers to discuss the published articles. It is an ethical responsibility to publish notes which criticize previously published articles, regardless of how painful it is for the publisher to publish them.
e.g. http://www.sciencemag.org/content/335/6069/665.summary
Reading an article that falsifies claims of another article is pure scientific joy. And such articles are not uncommon at all!
The problem is, as pointed out in the beginning of the paragraph "we need to get past the antiquated idea that the singular act of publication – or publication in a particular journal – should signal for all eternity that a paper is valid, let alone important".
While this process is currently conducted by paid-for editorial teams, most practitioners rely on citation tools which for any judgment provide links through to subsequent decisions which have, for example, "approved", "considered", "distinguished" (as in disapplied due to different facts, but this can have the practical effect of confining an earlier decision to its particular facts) or "overturned" that decision.
Although Google Scholar seems to provide some authority measure through the "cited by" tool, I've often wondered why academia (or perhaps I've just missed it) hasn't developed a comparable "precedent" system for research.
Dozens of flawed papers out of how many total papers? Numbers are meaningless without a context.
I agree. Will someone tell pg to turn off the upvotes?
There is absolutely no way that a researcher in that field can keep up with publications in all of those journals.
Also -1 for having the worst form of linkbaity title, which is one that is utterly false and misleading, and lying for the first two paragraphs of that blog post.
Also, he did make a better system. He co-founded the Public Library of Science (PLOS).
The author instead says that ideas of peer review need to change. Although, everyone I know always thought peer review was a continuous process, more support for after-publication review would be nice.
This tongue-in-cheek piece would be more plausible if the author could spell "phosphorus".
(https://en.wikipedia.org/wiki/Red_Phosphorous#Spelling_and_e...)
> According to the Oxford English Dictionary, the correct spelling of the element is phosphorus. The word phosphorous is the adjectival form of the P3+ valence: so, just as sulfur forms sulfurous and sulfuric compounds, phosphorus forms phosphorous compounds (e.g., phosphorous acid) and P5+ valence phosphoric compounds (e.g., phosphoric acids and phosphates).
I have: http://www.amazon.com/Confessions-Long-Distance-Sailor-Paul-...
If you had any idea to whom you're speaking, you would realize how utterly absurd your suggestion is. Speaking purely on a probabilistic basis, I know more about the world than you have any chance of knowing.
The bottom line -- in science, evidence means everything, reputation means nothing. The greatest amount of scientific eminence is trumped by the smallest amount of scientific evidence.
It's not like silicone / silicon - often misused where people have to guess from context. (No one is getting a silicon breast implant; no one is making silicone chips.)
It's not like mistaking the units, 1.50 mg instead of 1.50 μg, or putting the decimal point in the wrong place, 1.50 μg instead of 15.0 μg.
Having said all that the Underground Grammarian makes a good point in "Trifles":
> Our educators, panting after professionalism, are little interested in being known for a picayune concern with trifles like spelling and punctuation. They would much rather make the world a better place. They have tried on the gowns of philosophers, psychologists, and priests. That's why, when they think of their "teaching goals," they say those things in the questionnaire. They see themselves as guides to emotional development, instigators of creative capacities, and molders of moral character. When they must attend to the factual content of some subject, they prefer to say that they impart that "general insight into the knowledge of a discipline." Niggling details, like spelling and punctuation, seem base by contrast with those noble goals. Our educators have established for us what may be a genuinely new kind of cultural institution--although it is something like the Austro-Hungarian Empire--that stubbornly avoids those undertakings in which it might succeed and passionately embraces those in which it must probably fail.
(It's a long piece and I didn't know which bit to use as a quote). (http://www.sourcetext.com/grammarian/less-than-words-can-say...)
Sometimes true, but I've always thought it strange that people who spend much time with computers, and who know a horror story or two about seemingly trivial lexical errors resulting in catastrophes, object to those who correct spelling.
> It's not as if anyone is confused by the use of the incorrect word "phosphorous".
Not true. They're not even the same part of speech. The correct word is a noun, the misspelled word is (or can be) a verb.
In this case it went beyond a simple spelling error -- because the paper was technical and scientific, a different standard must be applied. In science publishing, how one spells technical terms is a deal-breaker. It is to science what "ingenious/ingenuous" means to a liberal arts graduate -- the difference is so great that it's a standing joke.
> It's not like silicone / silicon ...
Actually, to a biologist or a chemist, it's exactly like that.
> Having said all that the Underground Grammarian makes a good point in "Trifles" ...
I'm familiar with the piece and I agree with the sentiment, but this wasn't a trifle.
Writers regard this issue in the same way that musicians regard popular music. And scientists who publish must be writers as well as scientists.
The incorrect word is an adjective; it isn't a verb.