Mistakes Reviewers Make
sites.umiacs.umd.edu
sites.umiacs.umd.edu
Reviewers get no pay or remuneration, pauce guidelines, and no training (which is where articles like this can make a difference). There are no tangible career benefits for doing it (other than 'everyone else does it'), because you won't get any sort of official record for papers that you've reviewed. (And because it's single-blind, you'll never be credited on the paper.) There's very little feedback or quality control on reviews exerted by editors. You can't ever discuss the paper with your fellow reviewer(s). And it's an enormous time sink - reviewing a paper properly takes at least two hours, depending on the length and complexity. This is a real issue when you're in a field where doing lab research, writing your own grants and papers, reading the latest literature to keep up-to-date with the field, and possibly doing some teaching or admin, already takes up most of your time.
It's a seriously broken system. I inherently like the idea of doing reviews because it feels like you're giving back something to the community, but it ended up feeling like this good will was being taken advantage of by the journals, particularly the for-profit ones. I'm amazed that the whole system continues to work as well as it does.
"Detail-oriented: New researchers are often immersed in the minutiae of research, such as building software, collecting data, and running experiments. This means that they tend to focus on details (which may or may not be significant) rather than the bigger picture."
I am in my 40s, yet when I go to a job interview I am often interviewed by people in their 20s. I have 20 years experience with dozens of technologies. And yet, just recently, I found myself facing a long list of questions about the details of specific technologies, for instance, NodeJS. While I may not know the details about NodeJS, I had no trouble learning Struts and then Spring and then Ruby On Rails. Is there any reason to think I can't pick up the details of NodeJS? I have done one major project with Node, is it really crucial that I know all the latest packages before I get a job at your company?
In these interviews I am often surprised by the focus on very specific aspects of particular technologies. Who really cares? We all need to learn some new technologies for any job, even if it is just the specifics of the software that the company has built.
I am often surprised at the extent to which my 20 years of experience is discounted. However, I run into this less often when I am interviewed by someone who is in their 30s or 40s or 50s -- they seem more willing to recognize that I've had a long career and I've learned a lot of tech.
The reason that specific questions are asked, in my interview context anyway, is to ensure there is ideological alignment and acceptance of the technology. Whilst an experienced developer should be able to understand a programming language/framework, being willing to work with its idioms or recommended best practices can be difficult for some.
As an extreme example, if someone is well versed in procedural programming, and we work in an OO heavy environment, then my concern is about whether the candidate can understand the reasoning for our abstractions and contribute to our OO modelling conversations. The same could be said of an experienced OO practitioner being asked to code in a functional manner.
I think a good rebuttal, although I haven't had candidates do this nor I have tried this in interviews myself, is to to be well read about what a technological detail is about then draw parallels to what you do know in detail and explain how you are able to work with the technology the interviewer is talking about due to the similarities with your experience.
For example, working in a PHP shop, if I asked someone about Doctrine ORM and they responded that they have worked with Hibernate in Java and that allowed me to go into a line of questioning that discussed modelling and leads away from the specifics of Doctrine, I would be perfectly happy with such a response. On the other hand, if they outright dismiss it and say that they prefer to always right queries directly, I may need to question context and decide whether they are so inflexible about it that it would cause them issues when working with our codebase.
I've started to call these "Your baby is ugly" reviews.
The devil IS in the details. If a paper can't communicate the important details, then how can you ever claim that your work is reproducible? If your work is not reproducible, then it has no place in a scientific journal. In my field, a lot of senior researchers haven't executed a line of code in years, so the natural feeling is that the "detail" isn't important. If your code is not open-source and can't be audited, you better have the details in place.
Another thing that I think deserves A LOT more attention is the co-author list. There should be more in place to stop what's all too common: people forcing themselves, especially senior researchers, on to papers that they have no business being on. The setup in a lot of academic environments is such that this can't be tackled from the inside. I think this should be a fundamental part of a "reviewer manifesto": figure out who wrote the paper and if you can't, ask to find out.
In fact I would say an important "mistake reviewers make" is ... not actually doing much work. I've seen some appalling 2-3 line comments like "seems fine", even from senior academics. That's not even to talk about problems with misunderstood p-values, not reading the algorithm closely, not walking through the proof manually, and so on.