A Failure of Academic Quality Control [pdf]

I agree wholeheartedly. During my PhD, I did a big data compilation in my field (the fluid dynamics of certain spray nozzles), which ended up convincing me many commonly believed things in the field are false. I'm currently planning at least 3 paper on the issues I've found. This took a lot of time, but ultimately would save even more time and money if people were willing to fund it. I did the work while I was a teaching assistant and doubt many funding agencies would be willing to fund someone for a semester on this.

I'd like to be a professional reviewer. The closest existing job is a patent examiner, but they incentives there aren't great either. After my PhD I intend to at least do some consulting on the side, and perhaps "professional reviewer" could be a service I could offer. Worst case is that no one wants the service.

In this case the thing being criticized is a book, which is a somewhat different case. Reputable academic presses do send out book manuscripts for review, but it's a very different process from paper reviews, often involving only a partial early manuscript. I don't know how every press does it, but when I've reviewed for an academic press, I was given a draft of the first three chapters, and asked to give a recommendation for whether this was a serious academic book worth publishing, along with my positive and negative comments on the partial draft.

There are obviously some things I could usefully do there: it's often possible to identify and reject total crank books, to notice some early tendency towards overstating claims, to give general feedback on the direction of the book and whether it is likely to reach its audience, etc. But it's not the same as a journal, where you have the full paper to review and are expected to read it and give detailed feedback (which is also a more reasonable thing to ask of reviewers, because the text isn't as long!).

This reviewing approach is pretty traditional (it's how most famous philosophy books were published, for example) and relies heavily on good faith. Reviewers are supposed to judge whether this is a serious historian writing a good-faith book, but if they say yes, then you let the historian publish the book they want to publish without a lot of further rounds of vetting. It isn't expected they'll get everything right, but it's expected they'll make an earnest effort to get it right, perhaps even by employing research assistants themselves to do fact-checking. Anything they get wrong can then be corrected in the literature by someone else writing their own article or book in response.

One exception are books which are adaptations of dissertations. In that case there would have been more substantial review of the claims, by the dissertation committee, assuming nothing fishy happens between dissertation acceptance and adaptation into a book. The first author of the paper linked here (Lieberman) has a book out on the history of sex toys that's an adaptation of her dissertation, and tells a pretty different history than the one she criticizes in this paper. The book being criticized (by Maines) is not a dissertation adaptation, though.

> But then, Freeman's analysis is itself still controversial, so who knows.

Yeah it's always interesting as an outsider to read about settled scientific wisdom being enthusiastically challenged. I liked:

* the Gerta Keller dinosaur asteroid story -- https://www.theatlantic.com/magazine/archive/2018/09/dinosau...

* the Rebecca Fried "No Irish Need Apply" back-and-forth -- https://web.archive.org/web/20150805045521/http://intl-jsh.o...

Mathematics, for the most part. I wager this is due to the low material cost of verification.

The title is literally "A Failure ...". One single failure, and that's what it describes. There are only a handful of paragraphs about generalising, and these are knowingly subjective; two of them start with "We believe".

Probably augment, but not automate