PSU punishes prof who duped academic journal with hoax 'dog rape' article (2019)
campusreform.org
campusreform.org
The school asserted that Boghossian had unethically
conducted research on human subjects with his
experiment. According to the school’s Institutional
Review Board, Boghossian would have needed to obtain
“informed consent” from the individuals reviewing his
hoax articles in order for his actions to have been
considered ethical.
At what point does it become unethical simply to be a peer-reviewer for a broken journal? That's the implicit question that should be asked in this sort of ethics review. It does no good to punish just a submitter of papers, if the entire journal is unhealthy.
You can make the argument that there are too many low-impact journals and schools shouldn't be allocating resources towards keeping them going; I don't think it's a super strong argument but it's at least coherent. But more importantly: it won't have anything to do with this hoax; if you're willing to fake data, you can get accepted in higher-impact journals outside the social sciences as well.
More worryingly, data cannot model itself, so the quality of data should be largely irrelevant to any models which purport to explain it; as a result, a paper's contribution to science needs to be structured to work with other results, and cannot simply proclaim that its modelling is correct because some given equation fits the observed data. To pick a situation where your expertise shines, imagine that somebody submits a paper about a 3-SAT solver, and they include data that shows their solver doing not just extremely well on standard problem sets, but asymptotically scaling better than exponentially. The reason that you might doubt the honest presentation of the data is because you know, from having studied the field, that such behavior is unlikely.
[0] https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%...
"Who's Afraid Of Peer Review" targeted fee-charging open-access journals, which are financially incentivized to accept random papers. Journal reviewers are generally unpaid postgraduate academics.
What peer review does not say is "a panel of experts carefully vetted this paper to ensure that its conclusions are accurate". In fact, part of the premise of replication projects is that peer review doesn't say that.
I've only been a reviewer a couple times, and only in computer science. Other people on HN have experience reviewing for other hard science venues. Maybe their experiences are different. But I think it's notable that you don't hear that in comments about what this hoax exposed; has there been any HN comment from a reviewer saying that they were expected to rigorously vet submissions, the way a PhD board does with a thesis?
Daniel Engber makes many of these points better than I do, here: https://slate.com/technology/2018/10/grievance-studies-hoax-...
The message that peer review is largely meaningless hasn't really been made by academia, for obvious reasons, and thus stupid papers getting accepted by journals will continue to be interesting and news-worthy until people get the message that peer review doesn't mean much.
The world will then move on to "replicated" as a gold standard. This will be better but not by much. Just a few weeks ago Imperial College London published a press release claiming their Report 9 results from their COVID-19 simulator program had been replicated. Worth a press release because after it was open sourced it was discovered to be filled with non-deterministic behaviour, even with fixed RNG seeds.
Unfortunately the press release was fraudulent. The report they cited as evidence of replication was by a friendly academic. He said he was able to replicate the results, then admitted every number he got out was different, some by 10%-25% different. This is the output of a computer simulation so the allowable difference is 0%. Despite this, Nature and other outlets proceeded to report that ICL's COVID model was "replicated".
In the end, academia is strongly incentivised to appear credible, not actually be so. Being credible requires you can reliably turn your findings into something useful, as when in the realm of pure theory you can never be sure your findings actually hold up in reality. Corporate research has this attribute (eventually), academic research doesn't, as academics are rewarded for writing clever sounding papers not being correct.
The only difference is who's ox got gored.
Notably, the point of "Who's Afraid Of Peer Review" is that no reasonable reviewer could accept the paper for any reason; the papers (they were all practically identical) contained self-contradicting data; in fact, the plots in the paper were contradictory, so you couldn't even just skim the abstract and the graphics and then reasonably accept the paper. And, of course, the WAOPR papers were claiming to have cured cancer.
The WAOPR papers were designed to be trivial to spot. That's not the case for the S2 papers; the "dog park" paper, for instance, intricately describes 1000 hours of field work.
And, of course, there's the distinction that WAOPR papers targeted commercial services asking for money to publish papers, not volunteer peer review time.
If it was just the dog park paper, I could see your point, but come on, those papers are completely farcical.
It's not a Cache-22, it's plain old ultra-wokeness & wrongthink. Similar studies on "human subjects" (e.g. sending fake CVs to employers to spot biases in hiring) are lauded as exemplary and important just because they present socially-favourable results.
Social studies are one big hoax, most research is p-hacked & fake, and the rest is heavily politicized.
The motivation behind this is entirely reasonable. Bad "true negative" as well as known-good "true positive" articles ought to be regularly submitted to every peer review process as a test of accuracy. A great example of a similar idea was the 2014 NIPS consistency experiment [2].
However, there is a problem with the "dog rape" hoax article [3] as a test of the peer review system. Specifically, the author in that paper claims to have spent over 1000 hours carefully cataloguing the behavior of ten thousand people and dogs over the course of one year. The paper then proceeds to produce silly but surprising statistics: female dogs are 70% more likely to be leashed than male dogs, 100% of dogs with shock collars are male, 847 instances of dogs fighting were observed, and so on.
The problem here is that the data was falsified: the observations and experiments claimed never happened, the numbers were all made up. The article was likely accepted because of its data, but the data was a lie. If this data had been real, it might have been a minor but useful study relevant to economists, sociologists, or urban planners, regardless of how silly or made-up the conclusions at the end were.
Peer review works based on the assumption that the author is telling the truth about what experiments they conducted and what numbers they measured. The grievance studies authors could have submitted fake and silly hoax papers without falsifying data: that's what Sokal did, and that would have been a valid experiment testing the peer review system. But that is not what the authors chose to do.
[1] https://en.wikipedia.org/wiki/Grievance_studies_affair
[2] http://blog.mrtz.org/2014/12/15/the-nips-experiment.html
Not really.
They publish research articles. Depending on the field, and the field's philosophical perspective on the construction of knowledge, that can take differing forms. Those differing forms are not just valid but important. And understanding the differences is also, important. An article about medical treatment necessarily does different things and looks different than a journal in the humanities (for example). They make meaning from information in different ways - and that is what all forms of research do, make meaning. Research at its core is not about the discovery of facts...this is basic philosophy of science, basic Thomas Kuhn. Standards like replicability may be of use for making meaning in some fields, but they are not necessarily equally useful in others, often because the level of contextual situating that needs to occur reaches towards the impossible.
Serious members of the field would not have taken the Grievance Studies articles as 'capital-F-FACT' they instead would have interpreted it as a perspective, an argument, an interpretation. Specifically, the entire field of critical studies exists to, in varying forms, critique the ways in which 'discovery of facts' is a reductivist way of looking at meaning making that privileges certain perspectives over others - by treating some perspectives as reality.
So the critique here, and looking back at Sokal, isn't that 'hahaha I pulled one over on you' its that 'we assumed that you were giving us a new perspective in good faith that we could collectively learn from' and ...'now you are standing here laughing at us because you acted like a jerk.' It's two different world views with one willing to be open (by choice) to outside critique and perspectives, and the other supremely self-confident that critique and perspective is unnecessary.
In effect, it's like a legal opinion that separates matters of interpreting law and matters of finding fact.
Here is a more fair handed presentation of this case:
https://www.insidehighered.com/news/2019/01/08/author-recent...
The events I normally donate to support are canceled due to the pandemic, though, so I probably wouldn't donate anyway. Guess I can't really take a moral stance here.
Now that it's today, we're more vicious and this kind of punking will not go unpunished. Good checkpoint on where we're at.
It is inconceivable that someone with so little understanding of the toxic academic publishing environment could be influencing the education of 26,000 students.
Basically, Portland State University is retaliating against a professor who showed that the peer review and acceptance process for various journals was woefully broken.
The idea that academic journals are based on a presumption of good faith is totally alien to a lot of HN commenters.
The problem with the experiment, of course, is that reviewing takes a fuckload of time and effort, and most fields barely keep up with the legitimate workload they have. They are literally taking time and resources from program committees, and they do have IRB obligations in order to do that.
Out of the 21 papers submitted [1] 9 were rejected, 7 were published, 3 were asked to revised and resubmit for publication, and 1 was currently under review when the hoax was revealed. The fact that totally bogus papers have about a coin flip chance of being accepted is astounding.
Respectfully: I think what's shocking you is how many journals there are.
(Revise-and-resubmit, by the way, is a nice way of saying "reject").
At this point it seems like you're saying that peer review doesn't actually involve any sort of review. Fortunately, though, other academics don't share your experience that peer review is incapable of identifying faulty research. Because if it did, then there'd be little to no reason to put trust in academia.
We're saying the same things back and forth to each other at this point and can probably wrap it up.
If what you say is true, that reviewers don't bother reviewing the actual data, then mistakes like missing a decimal point and reporting figures an order of magnitude off would not be caught. That would be astounding, but fortunately most of my coworkers who have experience in academia do not corroborate your claim that reviewers don't bother to look at the data used to produce the paper.
What often happens is that, although the time they get to spend on a single paper is limited, reviewers still come up with important criticisms that end up leading to substantial changes (sometimes multiple rounds of them) or even an outright rejection.
I think the other poster summarizes why quite well why this charge of dishonesty is ironic. I'll just add a link to the paper itself if you'd like to read it [1], and review one part:
> From 10 June 2016, to 10 June 2017, I stationed myself on benches that were in central observational locations at three dog parks in Southeast Portland, Oregon. Observation sessions varied widely according to the day of the week and time of day. These, however, lasted a minimum of two and no more than 7 h and concluded by 7:30 pm (due to visibility). I did not conduct any observations in heavy rain. [...] The usual caveats of observational research also apply here. While I closely and respectfully examined the genitals of slightly fewer than ten thousand dogs [...]
So in the span of one year, this lone "researcher" claims to have "closely" inspected the genitals of ~10,000 dogs. That's 1,000 hours to inspect 10,000 dogs, which amounts to 10 dogs per hour, during which they took detailed notes on the dogs and owner's names, gender, and other associated information, while documenting the dogs' behaviour (6 minutes per dog+owner!). That stretches credulity to say the least.
Also, for the data to be meaningful, there must be at least 10,000 unique dogs visiting these three dog parks during the given time span. This also beggars belief even for Portland which features a high percentage of dog ownership. Portland has ~264,000 households, ~70% of households own a dog, that's ~185,000 dogs across ~32 dog parks, which is only 5,000 unique dogs per park on average.
The basic math just doesn't add up, and then the researcher disclaims their abilities to determine canine breeds, but makes claims like, "NB: the phrase ‘dog rape/humping incident’ documents only those incidents in which the activity appeared unwanted from my perspective – the humped dog having given no encouragement and apparently not enjoying the activity."
So apparently they have quite a bit of insight into canine behavioural psychology. There is a lot about the methods and the data that make no sense, and this paper received accolades.
> They are literally taking time and resources from program committees, and they do have IRB obligations in order to do that.
That's a legitimate concern. Unfortunately, the hoax itself reveals that these program committees may not be doing much meaningful work with those resources anyway, which seems like a far more important matter.
Edit: I would add that some way to verify that peer review is doing its job should be part of the publishing process. Periodic random hoaxes seem like a good way of doing it. It will make everyone, particularly reviewers, more skeptical and cautious.
> (Revise-and-resubmit, by the way, is a nice way of saying "reject").
No, it's a nice way of saying, "this is good work, you just need to massage your presentation".
(Here's a sharper way of asking the same question: tell me, as quickly as you can, how many dogs visit the largest Portland dog park; bear in mind that this is a waste of your time while you're tracking that stat down, because that's what the reviewer is thinking, too).
R&R means reject (it's a rejection cause). At Usenix, if I wanted you to "massage your presentation", I would accept conditional on those changes (actually: at Usenix WOOT, we would have assigned a reviewer to shepherd the paper --- we would have helped you massage your presentation).
Ultimately, to make a case that journals are accepting bad papers, you have to look at their accepts, not their rejects, no matter how those rejects are worded.
It seems to me that you're rejecting the entire premise of testing publications' abilities to detect fraudulent research.
This is like saying pen testing or red team exercises are fatally flawed. They're not. But they can definitely be embarrassing when they reveal deficiencies, so it's understanding why many would want to reject the results of these exercises. But if the organizations wants to improve they need to react constructively to the issues that were revealed, not dismiss the study as flawed. Unfortunately, in this situation the latter seems to be happening.
Of course, the purpose of the "Sokal Squared" "experiment" was to demonstrate how un-rigorous social science research is, but this is common in hard science fields as well, which just amplifies the dishonesty of the whole enterprise.
This is like a website saying that a vulnerability shouldn't be criticized because only bad faith actors would take advantage of it. Is that a reassuring response? Of course not. The whole point of being secure is to be secure from bad faith actors.
The whole point of Boghossian and the other researchers was to demonstrate how easily these publications can be taken advantage of by bad faith actors. Pointing out that these publications published these papers because the authors were bad faith actors is no excuse. How many other bad faith actors got fraudulent research published in these papers? We don't know, and we can't know. But we do know that these publications are extremely vulnerable to them.
It seems to me you aren't actually disagreeing with the conclusions made by these hoax papers: that these publications operate on a system of blind trust and can be easily exploited. It seems like you're saying that social science publications can't operate on any system other than blind trust, so we shouldn't think of this hoax as revealing anything significant. Personally, though, if these publications operate on a system of blind trust then that makes them inherently untrustworthy - hoax papers or not.
This thread is being rate limited, reply in edit:
At this point you're not even contending the claim that these publications fail to block bad research, and are instead claiming that other publications are just as ineffective. At this point you're accepting the thesis of these authors: these publications operate on blind faith and are incapable of identifying false papers.
If you submitted a paper solving NP hard problems in polynomial time it wouldn't be published - at least not without extensive scrutiny and checking if the most famous computer science question had indeed been answered.
Also, regarding the claim that papers don't receive feedback I suggest you read through the responses these authors of the hoax papers received. They did indeed receive feedback from their submissions, contrary to your comment, and some of it was truly astounding. Several praised the content but explicitly rejected the paper on the grounds of the race and gender of the author. In fact I'd say that the feedback was more important than the count of papers published.
To be specific: my experience is that reviewers allocate a very small amount of time to reviewing any particular paper, certainly not enough to reliably spot faked research results. Most of the work of reviewing is simply ranking papers, reducing an intractable set of submissions down to a tractable short list. Submitters are lucky to get any substantive feedback at all (hence the "reviewer #2" phenomenon).
If you have countervailing experience, describe it. Where have you reviewed, where you believe that program committee could withstand "red teaming"?
And I pointed out that this isn't a rebuttal to what these hoaxsters sought to reveal. In fact, it's an admission that their claims are true: These publication do operate on blind faith and are highly vulnerable to false claims.
You seem to be under the impression that peer review isn't meant to spot ineffective or false claims, and so this hoax reveals nothing. You're entitled to your own opinions, but many others disagree. Most understand that the purpose of peer review to ensure academic rigor and catch bad research, rather than operating under blind faith. So the revelation that many publications operate on blind faith is indeed a significant result.
In short, these hoaxsters are saying, "Look! These publications are incapable of spotting blatantly wrong research."
And you're responding, "But most publications can't spot blatantly wrong research.".
The second statement does not disprove the former. You're just claiming that the conclusions these hoaxsters made about social science publications can also be made against other publications. And that may be your perspective, but others do not have the same pessimistic attitude towards peer review.
The cite record in every field is littered with retractions, revisions, replication failures, and even outright fabrications.
At this point you're agreeing with the point these hoaxers set out to prove: these journals are poor at reviewing content and the content published in them should not be treated as having any level of authenticity or credibility.
> The cite record in every field is littered with retractions, revisions, replication failures, and even outright fabrications.
Again, at this point you're not even disagreeing with the claim the hoax paper authors are making. You're just saying that the same observations can be made in other fields.
1. https://www.vox.com/2014/11/21/7259207/scientific-paper-scam
Stefan is just saying that the same point the hoaxers are making can also be made towards other fields. That may be the case, but it doesn't make the hoaxer's statements any less true.
"This journal accepts blatantly bad submissions"
"But other journals also accept blatantly bad submissions"
The second sentence does not do anything to disprove the first.
It's the issue of saying he went to a dog park and observed and recorded dog behavior when he never did anything of the kind that really bothers me.
"no, you lied...we have processes for how to lie ethically, we told you about them and you ignored them"
"Showing you burglary is possible"
"I already knew that, and also no longer trust you as a neighbor"
'after getting caught black hat claims to actually be white hat'
i.e. any academic who doesn't think a paper should be published, but submits it on the off-chance they think it might be?