An Elaborate Academic-Journal Hoax
chronicle.com
chronicle.com
Yeah... that's the point! The whole point here is that you have these "conspicuously ideological" fields or subfields, which are not doing real scholarship, not interested in truth, but which rather are pushing a specific position and attempting to label as evil and shout down anyone who disagrees; and yet they're able to get away with claiming that they're doing real work, that this is what real scholarship is, and it's seriously damaging, especially (as the hoaxers note) to anyone who does want to do real truth-seeking work in these areas.
Sure, to those of us who realize what's going on, more hardly needs to be said. But lots of people don't, and the "grievance scholars" are slowly convincing the rest of academia that real scholarship consists not of neutral methods and honest argument, but agreeing with the grievance scholars in all things, and finding ways to inject it whether it's relevant or not. The point of this hoax is to make obvious what already should be.
To put it another way -- "grievance studies" is, as Dr. Steel says, an easy epistemological target. But it's a hard social target. It's built up a layer of protection around itself consisting not of truth but of social pressure and demands to conform or be ostracized. No claim, true or false, should be so protected, and that shield needs to be shattered. It's only an "easy target" in the imaginary world where people are rational and such transparent bullshit doesn't work. But in reality, it does, which is why there's a need to puncture it.
Rigor and importance are orthogonal concepts. Someone could do research of great rigor on a completely unimportant problem and people do bad research all the time on important ones.
Yeah, look- it's not that simple. In academia, we have to allow discussion and debate to be carried out freely, but obviously it doesn't make sense to dictate the subject of a debate and call it "free"! And the only people who are qualified to choose the subject of the debate are academics themselves- because nobody else has the expertise to make that decision.
So if academics choose to debate identity politics and "Theory"- well, that's what they get to debate. If the debate is done in a rotten manner that is not conducive to good scholarship, now that's another matter, but if the question is if those fields should exist, the answer is yes, and the reason is that they do (it's a tautology, OK?).
It's a feature, not a bug. It's a bit like in a democracy you must allow everyone their political opinion, even those people who are against democracy itself? In academia, we must allow all academics the ability to investigate the subjects of their interest, else we don't have academia, we just have a closed club of people who agree with each other- so, just like the fields targeted by the hoax.
Edit: If you like, you can replace my use of "fields and subfields" above with "departments and segments of such". That's probably fairer in other ways, too.
What degrees or disciplines would you characterise as "grievance scholars", and what evidence do you have that suggests the influence those disciplines have over scholarship is increasing?
The amount of degrees awarded in women's studies over the last 10 years has significantly dropped.
But that's not my argument at all? My argument is that these disciplines have destroyed the preconditions, the negative feedback loop, that keep a group of people truth-tracking, which includes norms of free inquiry and free argument, a basic understanding of what is and is not an argument (e.g. ad hominem and argument from consequences are not arguments), and have instead fallen into the positive feedback loop that results when these are removed, allowing tribal effects to dominate. I haven't said anything about lack of large-scale data. I mean, yes, large-scale data is important when possible, but it's hardly the most important thing. And indeed these fields often do have large-scale data, they just lack coherent arguments that this data supports their claim (data rarely bears directly on the claim being made, so you still need argument!).
This is simply, untrue. There are debates about both the semantics and broad claims in that specific case in these fields of study. I often find that criticisms of these fields lack a nuanced understanding of the actual content and behavior of the fields, and instead come from a malformed reading of the field that is readily accepted by the reader. Are there other specific beliefs in these fields that you're more concerned about?
That's not a false claim but rather a meaningless one; if people are debating that they have other problems. But, I guess, less serious ones; everyone makes meaningless statements occasionally, after all. One should avoid debating meaningless propositions, yes, but that's a mistake you can learn to avoid given good norms of discourse; getting those good norms of discourse is the more fundamental problem. (And of course it's not always obvious in advance what statements are meaningless; that's something that requires discussion to determine.)
Also, please don't put words in my mouth. Which you initially avoided doing, presenting your example purely as an example; but then you attempted to rebut this example as if this were an argument against my claims, which, obviously, makes no sense.
> I can make a guess at what thinkers you may be following
Would you care to make that explicit, since it sounds like you can?
In any case I'd be a little cautious of identifying me too closely with anyone established. If you want to know what I think more generally... well, you can find my writing scattered throughout the comments sections on such places as Slate Star Codex, Shtetl-Optimized, and Thing of Things. :P I've been meaning to write some of this up in a more visible form but, uh, I think I might want to get a new handle first...
> Which tribal effects do you think dominate?
Um, the usual ones? Do I really need to go over how tribalism works? Well, OK. Here are some characteristics, briefly:
Norms of free inquiry weaken and eventually vanish. Arguments are not evaluated by their reasoning, but by a combination of their conclusions (and their conclusions are evaluated by whether they support the tribal creed and the tribe's supremacy over its outgroups) and the social status within the tribe of the person making it. Everything becomes personal; principles are stated but ignored, in that they are applied selectively -- if a principle would be applied in a way that restrains someone in high standing in the group, or help someone in the outgroup, a reason for an exception is found. Agreeing with the tribe becomes considered as essential to personal ethics. Even just asking for clarification about a particular principle marks you as an evil outsider, because if you were a good person you would already understand. The group becomes taken over by those who only know how to politick and ceases to be about anything except its own dominance and the dominance of those who have taken it over. Group beliefs become more and more detached from reality.
I mean, I could go on, but I'm hoping that's sufficient.
This process is, of course, constantly happening everywhere, because it's human nature. But with a good negative-feedback loop, with good defenses against this, you can keep this positive-feedback loop in check, avoid the disastrous later stages, and (over the long term) keep yourself aligned with reality. Cut the negative-feedback loop and disaster is inevitable, even if its particular form is unpredictable.
("Tribalism" may be the wrong word here; I tend to use it as shorthand since these things tend to go together with tribalism proper. But the important thing is the phenomenon being described -- where the nominal purpose of a thing becomes completely corrupted by social considerations and interpersonal politicking -- not what we call it.)
We do have good defenses against this, if perhaps not on the mass scale. But certain people keep trying to toss out those defenses as unnecessary...
> Are there other specific beliefs in these fields that you're more concerned about?
I think this is a bad way of framing things. My concern here is with the meta-level, not the object-level. I'm not concerned not primarily about particular claims I see as wrong -- which, after all, may be correct (determining the truth of many of these claims does require large-scale data, something I'm not in a position to collect or trained in evaluating), but rather the norms of discourse surrounding them. Sure, a few might be correct by coincidence, but being reliably right requires good methods. Happening to get a few things right in your tribal creed is worth no credit.
To the extent I'm concerned with particular object-level claims it's primarily about how they're defended, about the reasoning given for them, not with the claims themselves. (The exception, of course, is those object-level beliefs that most directly justify this illiberalism and warped norms of discourse. I say "most directly" because oftentimes other claims are stated as justifications for this, but aren't actually doing the heavy lifting, or are even correct, and the real key assumption is left unstated.)
Or perhaps I should instead say, when it comes to particular claims, what concerns me is less often the claims that they accept than the claims they don't accept. I say "don't accept" rather than "reject" because they don't seem to explicitly reject these ideas but rather say things that contradict them while basically ignoring them and not engaging with them, despite their fundamental nature. I mean, OK -- these days you are starting to see more explicit rejection of the idea of free speech (for instance), rather than just attempting to ignore it or weasel around it. But there's a lot of just things like, people just ignoring the idea that espistemic rationality is essential to instrumental rationality, you know?
(And oftentimes there's the problem that what they're claiming may actually be a strategic equivocation. You've read "The Vacuity of Postmodernist Methodology", right? I won't bother repeating what Shackel has said better.)
> I often find that criticisms of these fields lack a nuanced understanding of the actual content and behavior of the fields, and instead come from a malformed reading of the field that is readily accepted by the reader.
So, OK. It's entirely possible that I've misidentified the problem, that it has less to do with these academic departments than I'd thought. But that's very different from there being no problem. The problem is very real and very visible. You don't need to go to the departments it's most famously associated with to see it -- you can see at Google; you can see it among Linux kernel developers; you can see it in the math department (my own field). It certainly does seem to me like it's being pushed by these departments and segments thereof it's most associated with, but, hey, I'm an outsider, maybe I'm mistaken there (although the tales I've heard of them are not encouraging). But the problem is apparent even if I am mistaken there and must be addressed all the same. Left unchecked it could seriously damage our ability to find correct answers and build new things (I mean, more than it already has).
But this is much more. This is about a set of ideas that pass for expert scholarship despite having lost any tethering to to the real world. Just look at this reaction to the hoax:
> Some of the pieces were indeed outlandish (and not all were accepted). But it appears that some were simply based in premises (e.g. social constructionism) or political principles (e.g trans equality) that the hoax authors find problematic but we do not.
https://twitter.com/alisonphipps/status/1047421076509261824
Note that word "problematic." I don't want to know whether social constructionism is considered "problematic", I want to know whether it is true.
There is an actual real world out there that scholars are purporting to study. But when science and empiricism are considered oppressive, all we are left with is posturing and grandstanding over ideology.
"...peer-reviewed journals are the absolute gold standard of knowledge production. And these concepts leak into culture... the problem of corrupt scholarship has already leaked heavily into other fields like education, social work, media, psychology, and sociology, among others—and it openly aims to continue spreading. This makes the problem a grave concern that’s rapidly undermining the legitimacy and reputations of universities, skewing politics, drowning out needed conversations, and pushing the culture war to ever more toxic and existential polarization. Further, it is affecting activism on behalf of women and racial and sexual minorities..."
https://areomagazine.com/2018/10/02/academic-grievance-studi...
The hoaxers writeup is, I think, pretty misleading. Their work tended to get rejected from high-impact journals. They submitted multiple stories to some journals (notably, to Hypatia, which they brag about getting accepted in) and leave a clear impression that their most ridiculous work was what made it in, rather than a much more banal paper. They quote almost entirely favorable feedback --- including, in their earlier writeup, from papers that were ultimately rejected! They brag about papers that were ultimately rejected. They submit approximately the same paper to multiple journals and are rejected by all but the very most marginal. In at least one case, they cite the same reviewer multiple times, leaving the impression of a consensus.
If you don't think that's misleading, it's worth knowing that Yascha Mounk, cited warmly in this piece, was clearly taken in, dazzled on Twitter by a rejected paper from the hoax.
What have they really uncovered?
1. Peer review isn't replication and never could have been. Peer reviewers get, at best, a couple hours, unpaid; submitting researchers get many months, and are often grant-funded. The hoaxers took ten months to get a few papers accepted in marginal journals. The pop understanding of what peer review is is, simply, broken: peer review depends on good faith, and it was never realistic to believe otherwise.
2. There are lots of crappy journals. Didn't you already know that? The market responds to the demand that professional academics publish, by creating venues that ensure professional academics can reliably publish. That's why we have measures like "impact factors". This is not at all a phenomenon isolated to the social sciences.
If the authors even acknowledged these conclusions, they'd take all the excitement out of their hoax. We'd all nod our heads and say, "of course". Instead, they pretend to have taken down something they've dubbed "grievance studies", while pretending that they couldn't have done the same thing to, say, a marginal economics journal.
> As for our performance, 80% of our papers overall went to full peer review, which keeps with the standard 10-20% of papers that are “desk rejected” without review at major journals across the field. We improved this ratio from 0% at first to 94.4% after a few months of experimenting with much more hoaxish papers.
This makes it sound like the authors wrote a bunch of absurd articles, all of which got desk rejected, and then they iterated until they got papers accepted somewhere. With that methodology, the conclusion ("we got papers accepted somewhere!") doesn't seem very surprising.
Edited to add: if the above is correct and their early submissions were all rejected, one could reasonably say of the results: "Editorial standards at all journals surveyed were more rigorous than the authors initially expected".
Give somebody a year and piles of submissions and of course they will sneak some things through.
What's your evidence for that? A number of their papers got accepted into decently high impact journals, including Hypatia.
> They submitted multiple stories to some journals (notably, to Hypatia, which they brag about getting accepted in) and leave a clear impression that their most ridiculous work was what made it in, rather than a much more banal paper
The thesis of the paper that got accepted was that "criticizing feminist theory is unethical". That doesn't seem particularly banal to me.
> They quote almost entirely favorable feedback --- including, in their earlier writeup, from papers that were ultimately rejected!
Who cares if the feedback was on papers that were ultimately rejected? Does that change the fact that they received favorable feedback on an absurd hypothesis?
> They submit approximately the same paper to multiple journals and are rejected by all but the very most marginal.
This is simply not true. They got accepted by high impact journals, including Hypatia. You seem to think that the papers that got accepted were 'banal', but that doesn't change the fact that they were accepted.
> In at least one case, they cite the same reviewer multiple times, leaving the impression of a consensus.
Are you saying that they do not cite a diverse sample of reviewers? Because that is empirically false. What you seem to be doing is latching a single instance where they cited the same reviewer twice, and trying to use that to insinuate that they didn't cite feedback from lots of distinct individuals, which is patently false.
> 1. Peer review isn't replication and never could have been. Peer reviewers get, at best, a couple hours, unpaid; submitting researchers get many months, and are often grant-funded. The hoaxers took ten months to get a few papers accepted in marginal journals. The pop understanding of what peer review is is, simply, broken: peer review depends on good faith, and it was never realistic to believe otherwise.
Is that so? So, you'd have no problem getting a nonsense paper accepted into a math, computer science, physics or even economics journal? I would be extremely surprised if you could pass of work of such low quality in journals of comparable impact factor in any of these disciplines.
For instance, the one they call "Hooters" is described as "A gender scholar goes to Hooters to try to figure out why it exists." The actual paper is written as a record of transcripts of visits to a restaurant and an extraction of particular conversation themes. At no point did I see the paper questioning why Hooters exists -- the (apparently totally faked) data seems pretty stereotypically motivated, which I think may be their point, but I'm not sure short of a fraud investigation how reviewers are supposed to know that the particular group this author claims to have visited Hooters with didn't say the things supposedly directly recorded. That's a really dramatic claim for a reviewer to make: "This conversation which the author claims is a direct transcription from a recording couldn't possibly have happened because it seems too stereotypical."
Yes, the methodology isn't that great, and the paper was rejected.
More generally, the "study" isn't selecting random journals to see if they could defraud -- it is instead aimed at specific academic targets and a "test to failure" scattershot mechanism is used. Thus we have no idea whether these journals are any more discriminating than, say, PLOS or Nature.
Ultimately, I had some sympathy for the Sokal experiment in that it seemed to say something about the interaction of literary theory and physics. I don't get the same sense that there's much here other than "confirmation bias exists, even among gender studies folks!!!" which seems like it wouldn't take 3 people 10 months to figure out, and could be done in a much more direct way, and honestly isn't that shocking or rattling a conclusion.
So did I. The dog park paper more or less matches its description. The fat bodybuilding paper is exactly as described. The dildos paper is as described. Do you have any descriptions aside from the Hooters one that you take issue with?
> Yes, the methodology isn't that great, and the paper was rejected.
Sure, but lots of them were not. I'm not sure why you would focus on a paper that was rejected when they had 7 that were accepted.
> More generally, the "study" isn't selecting random journals to see if they could defraud -- it is instead aimed at specific academic targets and a "test to failure" scattershot mechanism is used. Thus we have no idea whether these journals are any more discriminating than, say, PLOS or Nature.
True, though I have a hard time believing you could get the same result in any STEM discipline, though i'm certainly open to data to the contrary.
> Ultimately, I had some sympathy for the Sokal experiment in that it seemed to say something about the interaction of literary theory and physics. I don't get the same sense that there's much here other than "confirmation bias exists, even among gender studies folks!!!" which seems like it wouldn't take 3 people 10 months to figure out, and could be done in a much more direct way, and honestly isn't that shocking or rattling a conclusion.
I think the point of all this is that the epistemology of the field is intellectually bankrupt. It's not just confirmation bias. It's that within the epistemology of gender studies, they aren't even wrong.
The banality of the papers is the central weakness of the whole thing. The whole point of Sokal- or SCIgen-style hoaxes is that the papers were gibberish - so when a journal publishes one, the author can legitimately say "Hey look, this journal published gibberish!".
The authors here can't say anything like that. They wrote real papers, arguing real conclusions they thought the journals might agree with. Then those papers got rejected, so they toned it down and wrote more, and then they got peer-review feedback so they changed things in response, etc. The most one can fairly say about the results is "Hey look, these journals published papers that we think are really bad". Which is fair enough, but surely they could say the same thing about most of what those journals already publish, right?
The liberty of the whole thing is writing it all up as if a Sokal-style "we got them to print gibberish" narrative had occurred, when that seems clearly not to be the case.
My point is that most of them were not banal. In that line, I was responding to an isolated point of tptaceks. If all the papers were banal, i'd agree. But they were not.
> That dog parks are rape-condoning spaces and a place of rampant canine rape culture and systemic oppression against “the oppressed dog” through which human attitudes to both problems can be measured. This provides insight into training men out of the sexual violence and bigotry to which they are prone.
> That it is only oppressive cultural norms which make society regard the building of muscle rather than fat admirable and that bodybuilding and activism on behalf of the fat could be benefited by including fat bodies displayed in non-competitive ways.
> That it is suspicious that men rarely anally self-penetrate using sex toys, and that this is probably due to fear of being thought homosexual (“homohysteria”) and bigotry against trans people (transphobia). (It combines these ideas into a novel concept “transhysteria,” which was suggested by one of the paper’s peer reviewers.) Encouraging them to engage in receptive penetrative anal eroticism will decrease transphobia and increase feminist values.
> That men frequent “breasturants” like Hooters because they are nostalgic for patriarchal dominance and enjoy being able to order attractive women around. The environment that breastaurants provide for facilitating this encourages men to identify sexual objectification and sexual conquest, along with masculine toughness and male dominance, with “authentic masculinity.” The data are clearly nonsense and conclusions drawn from it are unwarranted by it. (NB. One reviewer did raise concerns about the rigor of the data)
> That academic hoaxes or other forms of satirical or ironic critique of social justice scholarship are unethical, characterized by ignorance and rooted in a desire to preserve privilege.
> No clear thesis. A rambling poetic monologue of a bitter, divorced feminist, much of which was produced by a teenage angst poetry generator before being edited into something slightly more “realistic” which is then interspersed with self-indulgent autoethnographical reflections on female sexuality and spirituality written entirely in slightly under six hours.
> That feminism which foregrounds individual choice and responsibility and female agency and strength can be countered by a feminism which unifies in solidarity around the victimhood of the most marginalized women in society.
I will grant you that 'breastaurants' and the Mein Kampf one are fairly banal. The Mein Kampf paper is really only shocking if you happen to know that it is an excerpt from Mein Kampf. But the rest seem pretty non-banal to me, at least with regard to being published in an academic journal.
I completely disagree. The point of this story is exactly that this stuff is line with what the journals normally publish. Their entire point is that the stuff they wrote is indistinguishable from these journal's normal fare.
Journals publishing papers indistinguishable from their normal fare is the null hypothesis! If accomplishing that was the authors' goal they could have taken ten months off, surely ;)
Please understand I'm not defending the content of these journals - I'm saying that this ten month hoax doesn't appear to have established anything.
This is your assumption. In fact they understood perfectly what the paper meant and that's why they published it. But because the paper was effectively constructed as something without meaning, Sokal managed to prove that their perception of meaning is a delusion.
Same thing has been done here: the editors of the journals found perfectly reasonable something that was ridiculous and absurd by construction, thus proving that the sense of reasonabless and soundness is lost to them.
I understand your point, but my argument here is that the hoax papers cannot reasonably be called "absurd by construction". The authors started out with that goal, but their initial papers apparently all got rejected so they iteratively toned down the absurdity until the papers got published. That's effectively "publishable by construction", isn't it?
Moreover, repeating it 20-fold? That's not even ethical - either one success proves your point, or were going to need a control group and rigorous methodology.
Since the trio reject the notion of control groups, they should have understood that getting more than one paper accepted was not furthering their experiment, only their ego.
My own comment leads off by acknowledging they got a paper accepted into Hypatia --- my gripe is that it's not the lurid farce they suggest they got accepted (a paper demanding that white male students sit on the floor in chains in class) but rather a much more boring paper about the legitimacy of hoaxes like theirs.
When I said "They submit approximately the same paper to multiple journals and are rejected by all but the very most marginal", I was referring to a specific paper (in fact, the Mein Kampf paper), not to all the papers. My point was: they tried that paper in several places, and the (academically) reputable venues rejected them.
Yes, I think --- I'm actually pretty confident --- that I could get a false paper accepted in a CS venue. Economists have already remarked on Twitter that you don't have to get "nonsense" papers accepted yourself, because the literature is already full of what turns out to be nonsense (do keep in mind that in large part what these hoaxers submitted was not nonsense, but rather fabricated research including field work).
IMO the paper they did get accepted is still quite disturbing.
> When I said "They submit approximately the same paper to multiple journals and are rejected by all but the very most marginal", I was referring to a specific paper (in fact, the Mein Kampf paper), not to all the papers. My point was: they tried that paper in several places, and the (academically) reputable venues rejected them.
Ok, but what about the others?
> Yes, I think --- I'm actually pretty confident --- that I could get a false paper accepted in a CS venue. Economists have already remarked on Twitter that you don't have to get "nonsense" papers accepted yourself, because the literature is already full of what turns out to be nonsense (do keep in mind that in large part what these hoaxers submitted was not nonsense, but rather fabricated research including field work).
False, sure. False papers get accepted all the time. That's part of the scientific process. These papers were not merely false, they were absurd. I seriously doubt you could get a truly absurd CS paper published anywhere remotely reputable.
What about medicine, genetics and neuroscience? These are the fields investigated by John Ioannidis, whom you've probably heard of [1]. He's the guy that published that paper titled "Why most published research findings are false" [2]. From the abstract of which, I quote:
Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.
Note that that's pretty much the claim of the hoax- that the hoaxed fields are dominated by bias and the majority of their research claims are absurd (so, false, I guess).
If that's true of medicine and genetics, it's true of most of the hard sciences, and no doubt, also CS and economics. So there is no good reason to single out gender studies or feminist philosophy for this kind of fault, or in any case, the hoaxers would have made a much more meaningful contribution if they had hoaxed a maths or physics journal, rather than the ones they did.
Btw- in maths, a recent HN article discussed the Mochizuki proof of the ABC conjecture, that his fellow maths experts consider to be very badly mistaken (if I got that right) [3].
_______________________
[1] https://en.wikipedia.org/wiki/John_Ioannidis#Research_findin...
[2] https://journals.plos.org/plosmedicine/article?id=10.1371/jo...
[3] https://www.quantamagazine.org/titans-of-mathematics-clash-o...
Again, the Mochizuiki situation is quite different. You're really not understanding these things if you think they're comparable. Mochizuiki's work is legitimate and real and serious. Whether or not it is correct is difficult to ascertain, because of its genuine complexity.
The accusation being made here is that these fields are incapable of separating absolute nonsense from genuine content. Which means that their work is indistinguishable from nonsense and absurdity. The work that Ioannidis did demonstrates that people acting within a good framework are producing work with a higher error rate than you might expect given the claims they make. That work is extremely important. But it doesn't invalidate the field of economics, or medicine. This work attempts to invalidate the entire edifice that these disciplines are built upon. It is extremely different and not comparable at all.
That the shoddy research investigated by Ioannidis is shoddy because it misuses statistics is besides the point. What matters is that the best efforts of peer- review in highly respected venues, in "hard" science fields fails to stop such work from being published.
Which makes the hoaxed fields (gender studies etc) not particularly special in that respect.
>> The accusation being made here is that these fields are incapable of separating absolute nonsense from genuine content.
That's also what Ioannides research concludes, except it does so for medicine, neuroscience, etc.
In Mochizuki's case, his proof was published even though experts in the field could not understand it and were not sure of its validity. It may yet prove to be nonsense, or, indeed- a hoax.
>> You're really misunderstanding Ionnandis' work if you think it is relevant to this.
Obviously, I believe it's you who doesn't understand what you're talking about. But perhaps we'll keep the discourse more civil by avoiding explicitly stated assumptions about each other's cognitive abilities?
Do you feel you are fallible in your identification of which fields are concerned with the truth and which ones aren't?
The replication crisis is a perfect example of the problem faced by all fields, of papers of questionable veracity being accepted by peer-reviewed journals. Convenient for you to say it is not up for discussion. However, I am casting a critical eye on your seemingly infallible nature, and I choose to consider it relevant.
You're absolutely right. Now, feel free to explain how that is relevant to THIS issue.
https://en.wikipedia.org/wiki/Replication_crisis
> A 2016 study in the journal Science found that two-thirds of 18 experimental studies from two top-tier economics journals (American Economic Review and the Quarterly Journal of Economics) successfully replicated. A 2017 study in the Economic Journal suggested that "the majority of the average effects in the empirical economics literature are exaggerated by a factor of at least 2 and at least one-third are exaggerated by a factor of 4 or more".
What this paper asserts is not that gender studies need to iterate on their techniques. It does not assert that the basic methodology of gender studies is good, but it needs some tweaks here and there. It asserts that, at a fundamental level, the concept of the field is flawed. They assert that the entire methodological framework is flawed. No one, not the authors of the replication crisis paper, or Ioannidis, nor any other serious academic asserts this about math, physics, etc...
Could it be perhaps that this individual paper, which you claim undermines the "entire methodological framework" of these fields, could be one of those individual studies that comes to incorrect conclusions, not because of any statistical flaw, but because of the inbuilt fallibility and political biases of the authors?
> If you presuppose that the category is absurd, the submission of an absurd paper is a vacuous act that proves nothing, and your point should be able to be easily demonstrated by your knowledge of the field as it stands.
It is my opinion and the thesis of the paper, that the papers that they got accepted to these journals is absurd. I will quote here the theses description of the 7 accepted papers:
Sure, most got rejected. But 7 got accepted. And while banality is obviously a subjective value judgment, I don't think it applies to all of them. Just to enumerate the hypotheses that were accepted:
> That dog parks are rape-condoning spaces and a place of rampant canine rape culture and systemic oppression against “the oppressed dog” through which human attitudes to both problems can be measured. This provides insight into training men out of the sexual violence and bigotry to which they are prone.
> That it is only oppressive cultural norms which make society regard the building of muscle rather than fat admirable and that bodybuilding and activism on behalf of the fat could be benefited by including fat bodies displayed in non-competitive ways.
> That it is suspicious that men rarely anally self-penetrate using sex toys, and that this is probably due to fear of being thought homosexual (“homohysteria”) and bigotry against trans people (transphobia). (It combines these ideas into a novel concept “transhysteria,” which was suggested by one of the paper’s peer reviewers.) Encouraging them to engage in receptive penetrative anal eroticism will decrease transphobia and increase feminist values.
> That men frequent “breasturants” like Hooters because they are nostalgic for patriarchal dominance and enjoy being able to order attractive women around. The environment that breastaurants provide for facilitating this encourages men to identify sexual objectification and sexual conquest, along with masculine toughness and male dominance, with “authentic masculinity.” The data are clearly nonsense and conclusions drawn from it are unwarranted by it. (NB. One reviewer did raise concerns about the rigor of the data)
> That academic hoaxes or other forms of satirical or ironic critique of social justice scholarship are unethical, characterized by ignorance and rooted in a desire to preserve privilege.
> No clear thesis. A rambling poetic monologue of a bitter, divorced feminist, much of which was produced by a teenage angst poetry generator before being edited into something slightly more “realistic” which is then interspersed with self-indulgent autoethnographical reflections on female sexuality and spirituality written entirely in slightly under six hours.
> That feminism which foregrounds individual choice and responsibility and female agency and strength can be countered by a feminism which unifies in solidarity around the victimhood of the most marginalized women in society.
I'll grant you that two of them, the 'breasturants' and Mein Kampf are relatively mild on the absurdity scale. The rest, in my view, are self-evidently ridiculous and/or unethical. If you disagree, feel free to explain why.
I'm really glad you're now saying that you just have the opinion that these papers and disciplines are absurd, because I'd consider that claim, stated as a fact, to be the realm of experts in the field. Presupposing the discipline is absurd would require you to have a deep knowledge of the field, which I'm not seeing much evidence of, here.
Where did I do that?
> I'm really glad you're now saying that you just have the opinion that these papers and disciplines are absurd, because I'd consider that claim, stated as a fact, to be the realm of experts in the field. Presupposing the discipline is absurd would require you to have a deep knowledge of the field, which I'm not seeing much evidence of, here.
You seem to have missed the point of their exercise. The point of their exercise is that the entire field is intellectually bankrupt. To say that the only people who can deem the field intellectually bankrupt are its members is fairly silly. Just read the papers. They're ridiculous. You've provided no defense whatsoever of their content.
You've come into this article, which only has evidence that papers written in bad faith can be accepted by peer-reviewed journals (trivial), already presupposing that the field is intellectually bankrupt. In order for me to believe you have the ability to make that judgement, I must also believe that you understand the field. People can be experts in fields without being a member of that field. You have fallen prey to the same false ideological purity you seem to have deemed yourself fit to criticize in others.
Since you're not advocating for rigorous empirical standards, I can only assume you feel the article we're commenting on was pretty pointless. And let's be real here, this article didn't effect your view either way. We're done here.
I'm not asking you to believe my judgment. Judge for yourself. Evaluate the content.
> Since you're not advocating for rigorous empirical standards, I can only assume you feel the article we're commenting on was pretty pointless. And let's be real here, this article didn't effect your view either way. We're done here.
I could say the same of you, without loss of generality. You could easily have tried to make the case that the papers here weren't absurd at all, but you didn't. You never tried to do that, you simply tried to obfuscate and hide. It's easier than making a real point, I guess.
In fact, you haven't even stated your position clearly in this entire discussion. You've carefully avoided actually saying "I don't believe these papers are absurd, and I do not believe the acceptance of them by these journals should make us at all skeptical of gender studies as a field". So, go ahead. State your position. I've stated mine.
It wasn't clear from the article, and those two categories have already done a number of hoax papers. So we already know that hoax papers get accepted occasionally. A new hoax study would only be interesting if you specifically at our to attack a field of study - at this point, there have been enough hoaxes that show feasibility overall.
Isn't the ability to separate out the quality work from the garbage the point of the peer review system? If the system was working as intended, the reviewers even with the assumption that they aren't being defrauded should have been able to sort the papers into the garbage pile.
There are two separate issues here. One is that peer review fails in disciplines that are largely bogus and based on mimicry and posturing.
IMO this absolutely applies to a lot of academic social sciences - but it also applies equally, allowing for changes in tone, convention and social register, to their ideological equivalents on the right.
The other is that we really have a crisis of authority and competence in general. Between fake news, fake reviews, more and less professional trolls and manipulators, empty academic tribal signalling, wilfully dishonest fraud in professional science, the entire PR and thinktank industry of spin, manipulation, and lies for influence, and the slightly milder but still toxic ad industry, there's far too little useful signal now among the tsunami of noise that we have to sift through every day.
If we expect reviewers to reject papers they find weird or suspicious then a lot of excellent scholarship will be rejected because established researchers are stuck in their belief systems. It also leads to serious problems when an author has obtained some hard-to-get data set and this is now a downside when it comes to review.
If the results are garbage then that will come out later when people work with the ideas presented in the paper.
The humanities (as opposed to the social sciences) are theory fields, they don't have experiment but they advance knowledge by good arguments that require real scholarship to support even if they don't have crispness of proofs. Someone reviewing it should go "Yep, that paper make a real claim and argues for it well enough that convincing counter-argument would need to be a significant work of scholarship in itself".
If a field of study produces a conversation made up of such arguments and counter-arguments, then it stands some chance of being growing repository of truth. Whereas if the field can't distinguish between it's own chatter and a hoax guided by mood affiliation, then the field is likely to be just chatter.
This is section really gets me. How in the world is science supposed to rely on “not-fraud”? Sure if you fake your data you might dupe a diligent reviewer, but this is why results are supposed to be reproduced.
Anything that relies on “not-fraud” seems inherently unscientific and straight up naive.
The problem lies in areas where replication and formalization is difficult. There is certainly knowledge to be gained in these areas. Though calling the fields sciences, when the scientific method cannot be effectively applied, seems dubious.
The solution is not obvious, but it is clear that steps should be taken to improve rigor in the less formal (straight forward) bodies of scholarship.
If a large percentage of scientists were to engage in deliberate fraud, there wouldn't be any feasible way to regulate that problem away. All available resources would end up being spent on scientists investigating each other for fraud, and no actual science would get done. Bear in mind that in many cases, establishing that an experiment is not fraudulent could require more time and effort than the original experiment!
As with many other human institutions, we have no choice but to rely on most people being honest most of the time.
If you want to reproduce an archaeology paper, well, how? You can’t exactly go dig the same site back up.
If you go out as a cultural anthropologist to hang out with some indigenous tribe somewhere, and your experiences aren’t the same as the last guy, does that mean anything? How much is because you’re different people? How much is because the tribe has already gotten used to having a researcher around?
This is part of why shit like the Piltdown Man happens.
If my banking API relies on user's not trying to commit fraud instead of proper validation, then I have a very insecure API. Making a statement that your system works as long as no one tries to break it sounds almost satirical.
The peer review system is just one step in the scientific process. After passing the review it gets published and more scientists see it. They can then tear it down or build on top of it. If it's actually bad science, it probably won't survive long.
Every few years someone will do a Sokal-hoax-esque thing in scientific fields, but usually they're just getting a paper into very-low-quality predatory journals that have only a nominal peer review process.
I'm wondering if that's what happened here. Do humanities fields even have the same set of incentives leading to a proliferation of those sorts of journals?
https://areomagazine.com/2018/10/02/academic-grievance-studi...
I'm not qualified to say if these truly were considered respectable journals in their fields but the authors had that as an explicit aim.
I looked up two journals they had published in and their articles got less than one citation per year typically ([1]&[2]), not exactly a booming field.
[1]: https://www.scimagojr.com/journalsearch.php?q=145138&tip=sid... [2]: https://www.scimagojr.com/journalsearch.php?q=14798&tip=sid&...
And let's hope it stays that way. This will help.
Everyone is distracted by the politics here when the real story is the effort.
Three people layman to these fields achieved a near tenure track publication list/ratio in a little over a year. I have a STEM PhD and I know that me and three of my friends couldn't do the same if we tried in a closely related STEM discipline much less an entirely different field.
Hypatia is the journal of record for feminist philosophy (and one of the few respected feminist journals in general). It should not be possible to go from 0 to published in a "real" top tier journal. If the paper is "not too crazy" that's even more damning! They produced a "real" paper as a joke? Do you really think that these same authors could get a paper published in The International Journal of Computer Vision in a little less than a year even if they were using fraudulent data? It would take them two years to know what they could and couldn't fake without being caught.
If the liberal arts don't want the sneering dismissal of their fields as low effort to become culturally entrenched they need to dump or reform the "X studies" departments and journals.
> "I am so utterly unimpressed," wrote Jacob T. Levy, a political theorist at McGill University, "by the fact that an enterprise that relies on a widespread presumption of not-fraud can be fooled some of the time by three people with Ph.D.s who spend 10 months deliberately trying to defraud it."
"Is there any idea so outlandish that it won’t be published in a Critical/PoMo/Identity/‘Theory’ journal?" tweeted the Harvard psychologist Steven Pinker.
"Three intrepid academics," wrote Yascha Mounk, an author and lecturer on government at Harvard, "just perpetrated a giant version of the Sokal Hoax, placing … fake papers in major academic journals. Call it Sokal Squared. The result is hilarious and delightful. It also showcases a serious problem with big parts of academia."
From what Pinker wrote, who knows if he even read it?
7⁄20 or 7⁄17, depending on how you want to count is concerning.
You could try working without any faith in your colleagues of course, but you are probably better of switching workplace if that sounds like reasonable behaviour to you.
You're mischaracterizing the system; it doesn't end with publication. After publication, the science goes out to a wider audience of scientists who can build on the paper or refute it if they find it's wrong. Sometimes this process can take a long time and many cycles of publicans, but that's just the nature of science.
We have to stop pretending like the peer review process is some kind of magical filter of true science, or that it was ever intended to be as such. That notion leads to the faulty idea that if it's published, it's true. Junk science gets published, even if it's not disingenuous, and that's probably true no matter how great you make the peer review system.
Explicit fraud isn't that common, since it does have risks to the submitter that far outweigh the (limited) benefits of getting an extra publication on your CV.
I've got a stack of rejected papers that beg to differ.
> Isn't the point that they serve as a basic filter for obvious noise?
Yeah, and obvious noise is usually rejected. But sometimes it is not so obvious what the noise is, and that's why the scientific process is bigger than the peer review system.
> Isn't the point that they serve as a basic filter for obvious noise?
Totally, there is a lot to fix about it. But if you really want to put your effort into changing the system, I don't think you should focus on peer review. Destroying the business model of journals is where I would start.
The purpose of peer review is to decide whether a paper is useful to the community, whether the presentation is good enough that the community can precisely understand what happened, and whether it has methodological issues that the authors may have missed. But we always approach reviews with a charitable lens, even when reviewing for extremely low-acceptance outlets.
The process that detects the bad research comes afterwards, as it should.
It clearly failed to do that in this case.
A system that prevents what the authors did here would necessarily reject a huge amount of useful research.
They can't verify that the data collected actually was collected without literally watching over the shoulder of the researcher while the researcher was collecting it, which is obviously not feasible. A researcher actively trying to deceive the reviewer with fake data but with a good method for collecting that data and analysis of the data that would be correct if the data were real is not something that should be caught by a reviewer. That should be caught by someone reproducing the research after publication.
I would submit that "fake research" is, in fact, bad research.
Instead I rely on the fact that the community will attempt to integrate these ideas after publication and arrive at the conclusion that the paper was garbage.
The Wikipedia article on peer review says "Peer review...occasionally detects fraud, but is not designed to do so."[1]
The article cites a Nature article that says "Peer review is not currently designed to detect deception, nor does it guarantee the validity of research findings. It should, however, identify flaws in the design, presentation, analysis and interpretation of science and provide prompt, detailed, constructive criticism to improve research."
[1] https://en.wikipedia.org/wiki/Scholarly_peer_review
[2] https://www.nature.com/nature/peerreview/debate/nature05007....
But is there really much to prove? What's the baseline? I mean, we're talking about a field where this got published - and it was not a hoax:
http://journals.sagepub.com/doi/full/10.1177/030913251562336...
The people behind this hoax had an unenviable task of matching and beating a paper like that (since, apparently, by itself it was not sufficient to prove the point). How would you do so?
I think there's some amount of feminist theory that boils down to -- all of the people thinking/doing stuff in A field were men operating under B culture; what if we thought about things from a different perspective & revisited some of our assumptions?
Like most fields, I bet glaciology has some underlying assumptions (is the goal of glaciology to manage glaciers? navigation?) & history (which glaciers have people historically studied? are they representative?)
And I know as an outsider, it's really hard to tell what's valuable about another field. If an outsider was critiquing computer science, I bet they'd easily be able to track down studies about tabs vs. spaces and laugh at the silly stuff that gets published in that field.
Here, for example, is a reference to a bad paper published in a field which is not usually the target of this kind of criticism:
https://academia.stackexchange.com/questions/9602/rediscover...
Besides, the paper in question wasn't merely published - the author got a prestigious NSF award that lists that paper:
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1253779
This, to me, implies that either such papers are considered broadly acceptable in the field, or they're not actually read. Not sure which is worse.
You're confused about how NSF grants work. The publication is listed as an output. The grant was awarded before that paper was published. The listing of the paper on that page implies nothing about its quality.
My example paper was torn down by mathematicians, not diabetes researchers.
SCIgen works like an academic "Mad Libs" of sorts, arbitrarily slotting in computer-science buzzwords like "distributed hash tables" and "Byzantine fault tolerance."
http://news.mit.edu/2015/how-three-mit-students-fooled-scien...
In April of 2005 the team’s submission, “Rooter: A Methodology for the Typical Unification of Access Points and Redundancy,” was accepted as a non-reviewed paper to the World Multiconference on Systemics, Cybernetics and Informatics (WMSCI), a conference that Krohn says is known for “being spammy and having loose standards.”
Not all legitimate scholarly work is based on a search for scientific truth. E.g. Interpretations/analysis of Shakespeare's work.
If the critics's objections could be distilled into something with a semblance of validity and germaneness, then it would be that the researchers's methods were too unfocused. But had they taken a narrower approach, the response could just have been to rationalize and ignore. Fight fire with fire (being generality, in this case).
They could, but the Bogdanov brothers did just that years earlier.
If you played this game long enough, undetected, could you get real scholars to cite your fake research?
Future work, perhaps.
(I don't have any direct evidence of this, but I strongly suspect this is an hypothesis that the authors would have wanted to test.)
In Sokal's single hoax in 1996 he (and his buddies, I guess) made a claim, followed it by a non-sequitur of true but uninteresting sentences about physics, then repeated the claim. It was a sandwich with no meat in it, and the (small, non-peer-reviewed) post-modern journal ate it up.
Sokal wanted to show that a certain style of post-modern writing substituted scientific jargon for an argument (or at least it used references to esoteric scientific fields as pretentious, uninteresting metaphors). The elegance of parodying this style was that his submission contained no data at all. It wasn't merely that the conclusion of the submission wouldn't be replicated. It was that any conscious human should notice the complete lack of supporting evidence for the claim!
The upshot of the journal accepting the article was that "the emperor had no clothes," so to speak. Sokal went on to write a book showing how several prominent post-modern thinkers used scientific non-sequiturs to make their non-scientific ideas sound more weighty. (Just as he had in his hoax.)
These three authors, however, are going a different route from Sokal-- specifically, one that provides fake data as evidence of a claim. (Or I guess they could have actually sat at dog parks for 1000 hours, but it doesn't matter for the point I'm making.) By providing fake data, this hoax passes an initial filter which the Sokal hoax did not-- namely, the filter that should reject submissions which lack any supporting evidence whatsoever. That means this hoax only reveals a lack of replication and/or verification, whereas the Sokal hoax reveals a lack of both peer review and basic reading comprehension even by a non-physicist. That makes this a weaker class of hoax than the Sokal hoax.
Finally, the authors purport to be going after an entire "ideological slant" in academia. Whatever one thinks of that aim, it is enormous and amorphous when compared to the narrow aim of Sokal keeping scientific jargon from being misused/misplaced.
So to recap:
* 1996: strong hoax successful against a single pomo journal, weak but persuasive claim against coopting scientific jargon for non-scientific aims
* 2018: weak hoax successful against 7 of (total number?) journals, strong and unpersuasive claim against all leftist academic fields
Now I'm curious-- did anyone in the late 90s criticize Sokal's hoax on the grounds that his careful constraints would be ignored by future pranksters?
Edit: clarification
What I hope has happened is the 20 revealed papers are not actually all of them and that they do a Wikileaks style drip release, wait on a response, then hit us with more much later.
Or weren't they aren't concerned about that type of ethics?
Or weren't they concerned about that type of ethics?
For these reasons, Sokal Hoaxes (and their like) are damaging wastes of time and the perpetrators should not be praised.
Shouldn't we be concerned with the possibility of undetected bad actors who are abusing it for more nefarious reasons?