Retraction of 60 articles implicated in a peer review and citation ring
uk.sagepub.com
uk.sagepub.com
"Doppelganger Finder: Taking Stylometry To The Underground" https://www.eecs.berkeley.edu/~sa499/papers/oakland2014-unde...
Code: Find multiple accounts (doppelgangers) of a user https://github.com/sheetal57/doppelganger-finder
https://psal.cs.drexel.edu/index.php/JStylo-Anonymouth https://github.com/psal/jstylo
Thinking "security through obscurity" is automatically and always bad is incorrect.
The people running this detection software don't do it to satisfy your intellectual curiosity. They do it because they are attempting to prevent attacks on their systems.
We also believe this particular fraud would have been identified regardless because we had a manual review scheduled for the most-cited articles of some of our journals, which would have unearthed this same thing.
In either case, fraud detection like this seems to be a fairly manual process for us at this point. Our short-term solution to ensure this doesn't happen again for this particular journal is to throw more editors at the problem. I'm unaware of what automated systems we currently have in place or plan to have in place, but that's something I'll be following up on.
If anyone happens to be working on a startup for automated fraud detection in academia I'm sure we'd love to talk :) As always, feel free to contact me directly, details in my HN profile.
Which is not to say the "publish or perish" problem isn't endemic, and that we can safely assume Taiwan is about as bad as any other "normal", non-cargo cult science country.
> that we can safely assume Taiwan is about as bad as any other "normal", non-cargo cult science country.
I wouldn't say that. For example, there's a strikingly consistent observation that close to 100% of acupuncture experiments succeed when they are done in East Asia (which is pretty much impossible for any effect size) despite rates halving when you move out of the region; is the placebo effect that much stronger in East Asia, or does this reflect investigator bias / incentives to get positive results by any means etc?
Do we know that China and Taiwan are worse? There are scandals in the U.S. too, such as papers funded by business interests without disclosure.
Many publications have looked into the problem:
[1] http://www.economist.com/news/china/21586845-flawed-system-j...
In 2010, however, Nature had also noted rising concerns about fraud
in Chinese research, reporting that in one Chinese government
survey, a third of more than 6,000 scientific researchers at six
leading institutions admitted to plagiarism, falsification or
fabrication.
Last month a collection of scientific journals published by
Zhejiang University in Hangzhou reignited the firestorm by
publicizing results from a 20-month experiment with software that
detects plagiarism. The software, called CrossCheck, rejected nearly
a third of all submissions on suspicion that the content was pirated
from previously published research. In some cases, more than 80
percent of a paper’s content was deemed unoriginal.
"Many agencies have toll-free hotlines and customer service
representatives working ten hours a day. Some even list the titles
of papers for sale, or the names of journals with which they
supposedly collaborate, on their websites. It's a sort of
Wild West of academic publishing. You have a whole laundry list
of services on offer that many scientists would probably never
imagine to be possible."What's interesting is that the sheer amount of fraud suggests that this thought process is considered at least somewhat normal by an entire society. Which societal factors could've led to this situation?
That said, it's probably an more inculturated process of what the fraud being discussed in the article.
In the way that Dr. Dre is in higher regard by the majority compared to Miles Davis and Miley Cyrus compared to Grace Hopper?
In the way that villagers aspire to be people who wear fancy suits and order people about for a living.
If "hapiness" was a criterium, drug dealers and pimps would be up their in the "achievement" scale.
I'm not saying the west doesn't have it's share, just that:
A) they are different shames and B) there's no magical rule which makes all cultures equally util. Some places/people are inevitably worse than others.
I remember someone lying about pretty much everything he bought, sometime making it nearly 10 times more expensive than it is (I've checked afterwards), probably to make himself seem wealthy.
Same in the office, it doesn't matter if you sit there all day doing nothing productive. As long as you leave late, it will make people think that you're working hard, and that's all that matters.
That's not to say that everyone in China is practicing the worst excesses, but even the most conciencious and moral people in China are put in a position where they have to manipulate the system just to get anywhere or pretty much do anything, even for basic licenses and permits. Everybody is on the same slippery slope whether they like it or not. It's just a question of how far you're willing to slide.
It's hard. I don't even have words to describe how hard that is. But it does at least stand a chance of working.
(This seems to be ground deeply into the human psyche. In general, a group of any kind is best off in the long term if the group polices itself for misconduct more aggressively than any outsider could. But the human instinct is to band together, assist each other to various degrees in covering over problems (often not even deliberately per se, but as a side effect of the system's structure), and band together to attack anyone external who notices anything amiss. This, alas, works in the short term quite well, but is death in the longterm.)
That's like saying "blacks are lazy" and when someone complains you tell them that it's normal for people to make such correlations and that to fix this he has to work on "making it not true".
You got it backwards. The burden of proof is on you.
You have to prove what you state with regards to corruption is true. Not just speak like it's a given, and demand of them to fix it.
>This seems to be ground deeply into the human psyche. (...) the human instinct is to band together, assist each other to various degrees in covering over problems (often not even deliberately per se, but as a side effect of the system's structure), and band together to attack anyone external who notices anything amiss.
This is a generalization that can be said for all groups. Not a proof (or even an indication) that any charge of "something amiss" from somebody external is right.
I have friends that studied in the UK (including Cambridge) and the US. My anecdotal evidence is that tons of papers in Comp-Sci in those countries are also BS, to keep the churn rate up). And academics return favors to other academics all the time, with regards to favorable reviews, etc.
(Btw, I'm not Chinese, so the pop psychology 101 above doesn't apply).
All I'm asking for is fairness. Too many of my colleagues have to disguise where they are from to even get by double blind review (there is a bias against China-based papers, and so we take careful care to extra-anonymyze our papers). This is ridiculous and immoral.
What you just 'notice' is anecdotal and biased by, um, your biases. It's not a correlation until you actually demonstrate it in a validly constructed analysis.
Of course, the trick is that legitimate attempts at validly constructed analyses are also effected by biases of the analyzers. When your conclusion is something that 'everyone knows' (but not neccesarily based on legit analyzed evidence), I think it's especially at risk of confirmation bias (when you try to muster the legit analyzed evidence).
It would be very interesting to find historical examples in the US of something that, for a while, scientists simply believed in, and believed that valid objective scientific experiments have shown to be so -- but then later the scientific community came to the opposite consensus. (ESP in the early 20th century might be one example, believe it or not. Look it up!) Science isn't quite as foolproof as one might like or assume.
At the moment, consensus science is that there isn't even such a valid object of inquiry as 'race', so you can't even make claims about it. (Although this is changing in interesting ways too, post-genomics)
Did you just do what you've been accusing others of? (Suggest a discriminated course of action based on a generalization of a certain nationality)
Unless you're talking about Aussie immigrants.
:)
Edit: isn't that obvious?
When it has merit, it is accepted as ethical.
It is ethical to deny a loan to people with a terrible credit score, because there is very good reason to suspect people with a terrible credit score will default. It's still discrimination, but it is ethical because it is based in fact.
Discrimination can have useful purpose. You can't expect people to expose themselves to a 9 out of 10 risk of being exploited by corrupt authors "in the name of equality".
But what's your alternative? Is racism such a Great Evil we are willing to risk corruption of the entire global scientific knowledge by not applying extra scrutiny to "high-risk" papers? We should allow fraudulent or corrupt papers to freely pollute the community, just to make sure everyone feels like they are being treated equally?
Frankly I care a lot more about good science than being politically correct. Besides, if your science is sound, why would you be upset if your paper undergoes extra scrutiny? Isn't that why you publish it, to be reviewed?
Please note that I'm not suggesting "all PRC scientists should be excommunicated from the scientific community" or anything like that. But trust has to be earned.
If you told me that "one out of three black American males will go to prison in their lifetime" and so, accordingly, you decided to be more suspicious of all of them, then you would receive a plethora of downvotes and would be (rightfully so) tagged as a racist. But since the generalization you made is about Chinese, so it is somehow more socially acceptable to say that you will treat their papers differently when you review them? Perhaps, but that is not objective and has more to do with the culture of the hackernews community.
> Frankly I care a lot more about good science than being politically correct. Besides, if your science is sound, why would you be upset if your paper undergoes extra scrutiny? Isn't that why you publish it, to be reviewed?
Paper reviewing is already very subjective, to add racism on top of that is just too much. Reviewers usually make up their minds about a paper in around 5 minutes, and then that bias clouds how they read the rest of the paper. My papers are usually immune to this because my name sounds "white," but my colleagues (who I often co-author with) receive no such benefit.
How exactly do you 'know' that (hypothetically) 9 out of 10 authors from Laos are corrupt?
How have you demonstrated/discovered this? (hypothetically, I realize this is just a hypothetical example). How sure are you that your findings weren't accidentally effected by biases, effecting your research design or approach to accidentally confirm your biases?
When I plotted percentage of debunked papers by country of origin, it was clear Elbonian papers are debunked 3x as often as anywhere else
With that kind of a wide net, I would think the main sampling errors would be things like "Only studied physics papers" or "forgot to gather data from Journal Y". The findings could be impacted, certainly, but hopefully still trustworthy enough to be actionable?
http://waset.org/publications/11354/applying-half-circle-fuz...
It sounds like a non-sequitur because it was probably copied and pasted from the grant proposal to get funding for the work. The acknowledgments show that funding came from the "National Science Council of the Republic of China, Taiwan", so that organization would naturally want to know why they're funding work on fuzzy numbers and how it benefits the country.
Academics are judged on stupid factors depending on the quantity of publications and citations rather than on the quality of their research.
The idea of bibliometrics is that you can compare on a single dimension individuals who do not work on the same subject, who use different methods, who do it in different communities which have different publication habits, etc. Of course this is completely stupid.
It's like saying that a cyclist who has participated in 10 Tour de France is better than a judoka who has participated to only 3 Olympic games. Only here you also give a job and funding to do it appropriately to the cyclist and say to the judoka to go f* himself until he participates in more Tour de France.
Of course, the thing is that you can't even have good criteria, because necessarily if you give criteria they become the goal, and instead of doing good research and publishing what is necessary when and where it makes sense, academics are forced to do research and have a publication policy that satisfy the arbitrary criteria. And even if the unlikely case where the criteria match what's better, it is impossible for it to be the case in the so many different fields of research.
The only good solution would be to judge academics by the two or three most relevant publications they have on the matter of the grant or job. That would require reading the publications, and if possible the report of the peer-review process (which would be enabled by open peer review).
If they had simply wanted to get published, then it seems like there are plenty of journals that will accept almost anything (they get paid on publication). More than 150 out of 300 open access journals accepted an obvious spoof paper last year[1].
Now, I would like to come back to the "article" you cite. Let me get this straight from the beginning: this text (by John Bohannon) is a piece of shit. Let me explain why:
The article aim to compare open access journals and closed access ones. The method that is used for that purpose is quite remarkable. It consists in sending a paper of very poor quality with wrong results in it to many gold open access journals (this means that there are what is called "Author Processing Charge": the authors pay the journal to publish), and to see how many of those will accept the paper. What happens is that a bit more than half of them accept the paper with, of course, not a single sign of peer-review happening. The author then conclude that open access journals are for the major part of poor quality, implying that they are worse than "traditional journals". This conclusion is eminently ridiculous.
First, if you want to compare open access and closed access journals, you also need to actually test the closed access journals. You can't just assume that they are good, especially when it's implied and not a single reason to think so is given (trust me it could be hard in many cases).
Second, the method makes no sense. No researchers sends his or her paper to unknown journals. When a paper is submitted to a journal, the journal is chosen depending on at least two criteria: on one side its prestige, because the more prestigious, the more read, and the goal of an academics is of course to be read by the other researchers in his domain; on the other side, for the seriousness of the journal (which is tacitly known in its academic community), because the peer-review process is very important for the authors (no academics like to have a paper published and then discover that there is a mistake in it). Amusingly, this is something that scientific news magazine such as Nature and Science are not that good at.
Third, the real conclusion of the poorly carried experiment that the author makes is that the bibliometrics pressure on researcher (“publish or perish”) coupled to their desirable and natural wish to be more widely read, and thus to publish in open access journals, gave birth to an unhealthy publication business that is harmful to research and science (I recall that the author only experimented with authors-pay open access journal, calling it simple "open access", as if he was, strangely, paid by publishers lobby to increase the already existing confusion between gold open access (where authors pay) and open access, which exists in many different models).
To me, the real solution to all these problem lies in Diamond Open Access and Open Peer-Review.
- http://en.wikipedia.org/wiki/H-index The big one. Used to judge researchers.
- http://en.wikipedia.org/wiki/G-index Another one for researchers. I haven't really come in contact with this one very much.
- http://en.wikipedia.org/wiki/I10-index Google's own little algorithm.
- http://en.wikipedia.org/wiki/Impact_factor Older. Used to judge journals
The first and last articles have fittingly long "criticism" sections. A key point to make here is that these bibliometrics succeed at their basic goal (providing one platform by which to judge all academics/publications) at the price of misaligning incentives.
Resultantly: you do get to judge all academics on an even platform, but that platform is a weighted average of how well they do research and how well they play the politics/popularity game, scaled by the popularity of their field (good luck finding me a researcher in theoretical plasma physics with an h-index over 30 -- I'm not sure there are even 30 theoretical plasma physicists in the US.).
On top of this, sites like ResearchGate (like LinkedIn, but for researchers) give people their own score, which is pretty opaque, and display it in bright green next to everyone's profile picture. It introduces a lot of competition to a field that doesn't really need it.
This reminds me of pagerank and SEO. In fact this ring is basically the equivalent of black-hat SEO's link-farms for academic journals.
leaves room
Also, for the back and forth links on Ioannidis's 2005 paper: https://en.wikipedia.org/wiki/John_P._A._Ioannidis#Research_...
It could be worth more than Bitcoin. I only wish that I already knew more about crypto so that I could both get it done correctly AND be the first solution on the market.
Research should not tend to satisfy arbitrary criteria. Nowadays with these h-index and impact factor crap, you see for instance journals which require (more or less officially) authors of an accepted paper to cite at least one or two other recent (1 or 2 years old max) papers from the journal.
Any reputation system will work only for its own benefit and not for the benefit of research.
I'm not at all saying that academics should not be evaluated. Only you can't judge the quality and pertinence of work at the state-of-the-art of so many domains with one (or even a few) number(s).
And how do you objectively tell which professors are better than others at doing research?
Whatever method you use, someone will try to game it.
Let the researcher decide. When an academic applies for a job he/she can attach his/her two or three most relevant papers.
> And how do you objectively tell which professors are better than others at doing research?
You can't, that's my point, the question makes no sense.
> Whatever method you use, someone will try to game it.
Then don't. Don't use a method. Evaluate a researcher by actually looking at her/his research.
Yes, it takes longer to read a paper than to read a number. But the number makes no sense and who decided that we have to be in a hurry when deciding how to spend mostly-public money on research and researchers?
That's actually a good question. What legitimacy have privately held journals, owned by publishers who sell back the results of research to the researchers who gave it to them for free (in the bast case scenario, sometimes you get to pay them to give it to them), to get this much influence on how to spend public money on research? Because that's pretty much what bibliometrics amounts to.
All you need to do is make the expected value of an attack against the system worth less than just doing some actual research.
you'll always have some people willing to take a gamble on the risk to get the reward, especially if it's a low % chance that they get any penalty at all. on top of the conscientious choice, humans are bad at estimating low-likelihood risks.