That wasn't the actual experiment, though. The actual experiment was to test whether or not these publications were capable of detecting fake papers.
The idea that academic journals are based on a presumption of good faith is totally alien to a lot of HN commenters.
The problem with the experiment, of course, is that reviewing takes a fuckload of time and effort, and most fields barely keep up with the legitimate workload they have. They are literally taking time and resources from program committees, and they do have IRB obligations in order to do that.
Out of the 21 papers submitted [1] 9 were rejected, 7 were published, 3 were asked to revised and resubmit for publication, and 1 was currently under review when the hoax was revealed. The fact that totally bogus papers have about a coin flip chance of being accepted is astounding.
Respectfully: I think what's shocking you is how many journals there are.
(Revise-and-resubmit, by the way, is a nice way of saying "reject").
At this point it seems like you're saying that peer review doesn't actually involve any sort of review. Fortunately, though, other academics don't share your experience that peer review is incapable of identifying faulty research. Because if it did, then there'd be little to no reason to put trust in academia.
We're saying the same things back and forth to each other at this point and can probably wrap it up.
If what you say is true, that reviewers don't bother reviewing the actual data, then mistakes like missing a decimal point and reporting figures an order of magnitude off would not be caught. That would be astounding, but fortunately most of my coworkers who have experience in academia do not corroborate your claim that reviewers don't bother to look at the data used to produce the paper.
What often happens is that, although the time they get to spend on a single paper is limited, reviewers still come up with important criticisms that end up leading to substantial changes (sometimes multiple rounds of them) or even an outright rejection.
I think the other poster summarizes why quite well why this charge of dishonesty is ironic. I'll just add a link to the paper itself if you'd like to read it [1], and review one part:
> From 10 June 2016, to 10 June 2017, I stationed myself on benches that were in central observational locations at three dog parks in Southeast Portland, Oregon. Observation sessions varied widely according to the day of the week and time of day. These, however, lasted a minimum of two and no more than 7 h and concluded by 7:30 pm (due to visibility). I did not conduct any observations in heavy rain. [...] The usual caveats of observational research also apply here. While I closely and respectfully examined the genitals of slightly fewer than ten thousand dogs [...]
So in the span of one year, this lone "researcher" claims to have "closely" inspected the genitals of ~10,000 dogs. That's 1,000 hours to inspect 10,000 dogs, which amounts to 10 dogs per hour, during which they took detailed notes on the dogs and owner's names, gender, and other associated information, while documenting the dogs' behaviour (6 minutes per dog+owner!). That stretches credulity to say the least.
Also, for the data to be meaningful, there must be at least 10,000 unique dogs visiting these three dog parks during the given time span. This also beggars belief even for Portland which features a high percentage of dog ownership. Portland has ~264,000 households, ~70% of households own a dog, that's ~185,000 dogs across ~32 dog parks, which is only 5,000 unique dogs per park on average.
The basic math just doesn't add up, and then the researcher disclaims their abilities to determine canine breeds, but makes claims like, "NB: the phrase ‘dog rape/humping incident’ documents only those incidents in which the activity appeared unwanted from my perspective – the humped dog having given no encouragement and apparently not enjoying the activity."
So apparently they have quite a bit of insight into canine behavioural psychology. There is a lot about the methods and the data that make no sense, and this paper received accolades.
> They are literally taking time and resources from program committees, and they do have IRB obligations in order to do that.
That's a legitimate concern. Unfortunately, the hoax itself reveals that these program committees may not be doing much meaningful work with those resources anyway, which seems like a far more important matter.
Edit: I would add that some way to verify that peer review is doing its job should be part of the publishing process. Periodic random hoaxes seem like a good way of doing it. It will make everyone, particularly reviewers, more skeptical and cautious.
> (Revise-and-resubmit, by the way, is a nice way of saying "reject").
No, it's a nice way of saying, "this is good work, you just need to massage your presentation".
(Here's a sharper way of asking the same question: tell me, as quickly as you can, how many dogs visit the largest Portland dog park; bear in mind that this is a waste of your time while you're tracking that stat down, because that's what the reviewer is thinking, too).
R&R means reject (it's a rejection cause). At Usenix, if I wanted you to "massage your presentation", I would accept conditional on those changes (actually: at Usenix WOOT, we would have assigned a reviewer to shepherd the paper --- we would have helped you massage your presentation).
Ultimately, to make a case that journals are accepting bad papers, you have to look at their accepts, not their rejects, no matter how those rejects are worded.
It seems to me that you're rejecting the entire premise of testing publications' abilities to detect fraudulent research.
This is like saying pen testing or red team exercises are fatally flawed. They're not. But they can definitely be embarrassing when they reveal deficiencies, so it's understanding why many would want to reject the results of these exercises. But if the organizations wants to improve they need to react constructively to the issues that were revealed, not dismiss the study as flawed. Unfortunately, in this situation the latter seems to be happening.
Of course, the purpose of the "Sokal Squared" "experiment" was to demonstrate how un-rigorous social science research is, but this is common in hard science fields as well, which just amplifies the dishonesty of the whole enterprise.
This is like a website saying that a vulnerability shouldn't be criticized because only bad faith actors would take advantage of it. Is that a reassuring response? Of course not. The whole point of being secure is to be secure from bad faith actors.
The whole point of Boghossian and the other researchers was to demonstrate how easily these publications can be taken advantage of by bad faith actors. Pointing out that these publications published these papers because the authors were bad faith actors is no excuse. How many other bad faith actors got fraudulent research published in these papers? We don't know, and we can't know. But we do know that these publications are extremely vulnerable to them.
It seems to me you aren't actually disagreeing with the conclusions made by these hoax papers: that these publications operate on a system of blind trust and can be easily exploited. It seems like you're saying that social science publications can't operate on any system other than blind trust, so we shouldn't think of this hoax as revealing anything significant. Personally, though, if these publications operate on a system of blind trust then that makes them inherently untrustworthy - hoax papers or not.
This thread is being rate limited, reply in edit:
At this point you're not even contending the claim that these publications fail to block bad research, and are instead claiming that other publications are just as ineffective. At this point you're accepting the thesis of these authors: these publications operate on blind faith and are incapable of identifying false papers.
If you submitted a paper solving NP hard problems in polynomial time it wouldn't be published - at least not without extensive scrutiny and checking if the most famous computer science question had indeed been answered.
Also, regarding the claim that papers don't receive feedback I suggest you read through the responses these authors of the hoax papers received. They did indeed receive feedback from their submissions, contrary to your comment, and some of it was truly astounding. Several praised the content but explicitly rejected the paper on the grounds of the race and gender of the author. In fact I'd say that the feedback was more important than the count of papers published.
To be specific: my experience is that reviewers allocate a very small amount of time to reviewing any particular paper, certainly not enough to reliably spot faked research results. Most of the work of reviewing is simply ranking papers, reducing an intractable set of submissions down to a tractable short list. Submitters are lucky to get any substantive feedback at all (hence the "reviewer #2" phenomenon).
If you have countervailing experience, describe it. Where have you reviewed, where you believe that program committee could withstand "red teaming"?
And I pointed out that this isn't a rebuttal to what these hoaxsters sought to reveal. In fact, it's an admission that their claims are true: These publication do operate on blind faith and are highly vulnerable to false claims.
You seem to be under the impression that peer review isn't meant to spot ineffective or false claims, and so this hoax reveals nothing. You're entitled to your own opinions, but many others disagree. Most understand that the purpose of peer review to ensure academic rigor and catch bad research, rather than operating under blind faith. So the revelation that many publications operate on blind faith is indeed a significant result.
In short, these hoaxsters are saying, "Look! These publications are incapable of spotting blatantly wrong research."
And you're responding, "But most publications can't spot blatantly wrong research.".
The second statement does not disprove the former. You're just claiming that the conclusions these hoaxsters made about social science publications can also be made against other publications. And that may be your perspective, but others do not have the same pessimistic attitude towards peer review.
The cite record in every field is littered with retractions, revisions, replication failures, and even outright fabrications.
At this point you're agreeing with the point these hoaxers set out to prove: these journals are poor at reviewing content and the content published in them should not be treated as having any level of authenticity or credibility.
> The cite record in every field is littered with retractions, revisions, replication failures, and even outright fabrications.
Again, at this point you're not even disagreeing with the claim the hoax paper authors are making. You're just saying that the same observations can be made in other fields.
1. https://www.vox.com/2014/11/21/7259207/scientific-paper-scam
Stefan is just saying that the same point the hoaxers are making can also be made towards other fields. That may be the case, but it doesn't make the hoaxer's statements any less true.
"This journal accepts blatantly bad submissions"
"But other journals also accept blatantly bad submissions"
The second sentence does not do anything to disprove the first.
It's the issue of saying he went to a dog park and observed and recorded dog behavior when he never did anything of the kind that really bothers me.
"no, you lied...we have processes for how to lie ethically, we told you about them and you ignored them"
"Showing you burglary is possible"
"I already knew that, and also no longer trust you as a neighbor"
'after getting caught black hat claims to actually be white hat'