Reasonable critics aren't expecting reviewers to replicate the results, but the dog park paper illustrates that reviewers can't even recognized clearly bogus data. I mean, have you even read the methods the paper said they employed?
The problem with the experiment, of course, is that reviewing takes a fuckload of time and effort, and most fields barely keep up with the legitimate workload they have. They are literally taking time and resources from program committees, and they do have IRB obligations in order to do that.
Out of the 21 papers submitted [1] 9 were rejected, 7 were published, 3 were asked to revised and resubmit for publication, and 1 was currently under review when the hoax was revealed. The fact that totally bogus papers have about a coin flip chance of being accepted is astounding.
Respectfully: I think what's shocking you is how many journals there are.
(Revise-and-resubmit, by the way, is a nice way of saying "reject").
At this point it seems like you're saying that peer review doesn't actually involve any sort of review. Fortunately, though, other academics don't share your experience that peer review is incapable of identifying faulty research. Because if it did, then there'd be little to no reason to put trust in academia.
We're saying the same things back and forth to each other at this point and can probably wrap it up.
If what you say is true, that reviewers don't bother reviewing the actual data, then mistakes like missing a decimal point and reporting figures an order of magnitude off would not be caught. That would be astounding, but fortunately most of my coworkers who have experience in academia do not corroborate your claim that reviewers don't bother to look at the data used to produce the paper.
What often happens is that, although the time they get to spend on a single paper is limited, reviewers still come up with important criticisms that end up leading to substantial changes (sometimes multiple rounds of them) or even an outright rejection.
I think the other poster summarizes why quite well why this charge of dishonesty is ironic. I'll just add a link to the paper itself if you'd like to read it [1], and review one part:
> From 10 June 2016, to 10 June 2017, I stationed myself on benches that were in central observational locations at three dog parks in Southeast Portland, Oregon. Observation sessions varied widely according to the day of the week and time of day. These, however, lasted a minimum of two and no more than 7 h and concluded by 7:30 pm (due to visibility). I did not conduct any observations in heavy rain. [...] The usual caveats of observational research also apply here. While I closely and respectfully examined the genitals of slightly fewer than ten thousand dogs [...]
So in the span of one year, this lone "researcher" claims to have "closely" inspected the genitals of ~10,000 dogs. That's 1,000 hours to inspect 10,000 dogs, which amounts to 10 dogs per hour, during which they took detailed notes on the dogs and owner's names, gender, and other associated information, while documenting the dogs' behaviour (6 minutes per dog+owner!). That stretches credulity to say the least.
Also, for the data to be meaningful, there must be at least 10,000 unique dogs visiting these three dog parks during the given time span. This also beggars belief even for Portland which features a high percentage of dog ownership. Portland has ~264,000 households, ~70% of households own a dog, that's ~185,000 dogs across ~32 dog parks, which is only 5,000 unique dogs per park on average.
The basic math just doesn't add up, and then the researcher disclaims their abilities to determine canine breeds, but makes claims like, "NB: the phrase ‘dog rape/humping incident’ documents only those incidents in which the activity appeared unwanted from my perspective – the humped dog having given no encouragement and apparently not enjoying the activity."
So apparently they have quite a bit of insight into canine behavioural psychology. There is a lot about the methods and the data that make no sense, and this paper received accolades.
> They are literally taking time and resources from program committees, and they do have IRB obligations in order to do that.
That's a legitimate concern. Unfortunately, the hoax itself reveals that these program committees may not be doing much meaningful work with those resources anyway, which seems like a far more important matter.
Edit: I would add that some way to verify that peer review is doing its job should be part of the publishing process. Periodic random hoaxes seem like a good way of doing it. It will make everyone, particularly reviewers, more skeptical and cautious.
> (Revise-and-resubmit, by the way, is a nice way of saying "reject").
No, it's a nice way of saying, "this is good work, you just need to massage your presentation".
(Here's a sharper way of asking the same question: tell me, as quickly as you can, how many dogs visit the largest Portland dog park; bear in mind that this is a waste of your time while you're tracking that stat down, because that's what the reviewer is thinking, too).
R&R means reject (it's a rejection cause). At Usenix, if I wanted you to "massage your presentation", I would accept conditional on those changes (actually: at Usenix WOOT, we would have assigned a reviewer to shepherd the paper --- we would have helped you massage your presentation).
Ultimately, to make a case that journals are accepting bad papers, you have to look at their accepts, not their rejects, no matter how those rejects are worded.