While I agree that there are systemic problems with peer review and how the science "enterprise" works, there is fitting analogy from politics by Winston Churchill: "The worst form of government except for all the others."
While I agree that there are systemic problems with peer review and how the science "enterprise" works, there is fitting analogy from politics by Winston Churchill: "The worst form of government except for all the others."
A NLP-based referencing system would not need to be correct all the time; it would merely need to be helpfully suggestive. As you're writing your paper, it would put tips in the sidebar: "Maybe this is relavent? (Hover to read abstract)". As long as there aren't an intolerable number of false positives, it would be quite a useful tool, I think.
Come to think of it that this is probably the most important detractor for having an NLP based "recommender." Personally, something like this might be interesting, probably even a great help, but at the end of the day, people need to really read a lot of papers, follow the proceedings of their target conferences, journals, and asking colleagues for their bibliographies. This has the added benefit of teaching them how to present their own work in contrast to others, do meaningful evaluations (in the best of all worlds, of course!) and figure out who is doing interesting work and might be valuable to get into contact with. Of course, some parts could be automated, but there is currently no incentive for scientists to do so.
IMHO, it would be a much more important step for CS researches to publish their code, too, because I frequently come across papers that have no implementation or evaluation at all--and that's really bad, because then the least-publishable unit becomes an idea with nice pictures. Researchers can be very successful using this "publication strategy." Come to think of it, there should be another approach to rank scientists by the number of publications, or their impact; unfortunately, I have no idea what could work instead.
Not really, because other researchers are advancing their careers based on how often and how much they (a) publish and (b) get cited. So the colleagues most likely to be in a position to review your work are those who got cited a lot, who would primarily know about their work that got cited a lot.
The application of a relevance/NLP/PageRank-like additional-citation recommender program could come as a step in the reviewing process. Rather than having just human reviewers suggest further reading, a "machine reviewer" would as well, placing the query results in front of everyone involved in publishing a paper.
We need to try more.
When you are venturing into the unknown, innovation is tough and challenging, but it's also very rewarding.
But I agree - we should experiment more. One idea - we should be A/B testing any new law, before introducing it in the whole country.
Usually only with the forms of government that don't care about millions of victims. Is really a no-brainer that any totalitarian system that intends to make an enemy of a large section of its population is probably going to be pretty awful. So I would strongly suggest weeding those forms out in advance, rather than using the common historical method of allowing them to be implemented through force by the paranoid and insane.
This is less of an issue when it comes to systems of improving feedback in science, unless it somehow enables an aspiring evil genius, or something.