“…because conferences have made it mandatory to review 4-5 papers if you submit to them.”
Is that really a thing? So if I submit a legit paper it is being reviewed forcefully by a bunch of random people from god knows where?
Is that really a thing? So if I submit a legit paper it is being reviewed forcefully by a bunch of random people from god knows where?
I was forced to review 7 papers.
Now, how are these papers chosen for you?
You get to bid on which to review.
Bid on 30 papers out of 30,000.
I got none of those, and I had to review 7 papers I wasn't really competent enough to review.
I went to smaller conferences, I suppose, but back then I also had to review papers outside my direct expertise, although nothing too remote. But I didn't know the literature well, obviously. I was able to weed out the sub-par papers (I think), but it was harder to estimate the merits of those that did make sense, as I only saw 1 or 2 per area. And that's what determines the acceptance, after all. No criticism because the reviewer judged it perfect or because the reviewer didn't know what to look for? Outcome is the same.
It doesn't scale.
So basically leave it up to the Area Chair. It still helps them a bit that you give the paper a quick once over and flag it up if it's somehow complete crap. The problem is that I'm probably missing the chance to review papers I'd be really interested in and that I could help to improve.
I guess it's a bit ridiculous that AI conferences in particular can't find a systematic way to route papers to subject matter experts. I guess the usual system with area and sub-area keywords is overwhelmed by the extreme increase in numbers. This year AAAI made link to my Google Scholar and DBLP, link my top five papers, and add some more info about my subject to my profile; and they still failed to rate the papers that were really relevant as most relevant. At least the bidding pool was a bit smaller this time so I guess they're trying to work something out.
Meanwhile submission numbers are exploding and I fear all the conferences can do is try to play catch up.
Another idea: make it less about the written word and more about artifacts. Require more proof, more protocols, more spreadsheets, more code, etc.
And stop caring about the words so much, as they really don’t mean anything anymore. Larger papers should be punished instead of rewarded; the shorter the paper (in terms of words) the higher the chance to be published. Every word should be backed up by data, code, etc. Any paragraph that doesn’t pair with proof should mean rejection.
As for it "being a thing": sadly yes. The number of papers has been steadily increasing over the years and the number of volunteer authors is simply not enough.