Real peer review has never been tried
worksinprogress.co
worksinprogress.co
- integration of preprint servers and alt metrics
- tweaking incentives to review
- making comments on papers public
- use of software to detect fraud
- directing resources specifically to improving peer review
The bigger problem is that the author doesn't seem to actually zero in on the problem peer review is supposed to solve today. The author notes that peer review really got going in the 1970s as a way to filter content flowing to overwhelmed editors. But the emergence of the internet largely nullifies that problem. Wide distribution of scientific information no longer requires scientific publishers.
The real problem is the ways in which science funding, journals, and peer review have become intertwined, with publishers playing the role of bankers in this economy. This problem is cultural, not technical. It's a historical relic and it increasingly does not serve science well.
So, what is the actual problem that journal-supervised peer review is supposed to solve in the age of the internet?
The decisionmakers do have enough time to learn which are the most prestigious journals in the field. So, they can pick the people with the most papers submitted to prestigious journals, or at least use that to filter applicants down to a short list for closer examination.
It's also pretty devastating evidence that the world is not going to improve in any sort of an organized way if the experts that we would expect to lead the effort for a rational world can't clean their own house. It's hard to trust academic systems to design ways to improve society when the academic system is built around an irrational base in journals.
An academic system that exhibits the same shitty array of characteristics as every other corrupt status quo institution doesn't give me a lot of hope for everything else.
Wide and voluminous distribution of bad information requires filters to extract the good, peer review has some form of filtering functionality, although I wouldn't say it is great I think it would probably be better than the filter that a Facebook or Twitter of Science would provide (or just Facebook or Twitter if you don't like the 'of science' locution)
I'm not at all convinced that this is true - people have been saying it for a long time, and it's not manifested itself in a particularly compelling fashion yet.
The publishing industry survives because researchers need to put "Published in Journal X" on their CVs. The peer review process also can lead to incremental improvements, and will occasionally catch major errors, but at this point everything other than the stamp of approval is secondary.
Beyond that, their curation is...lackluster at best, even if we're not talking about gatekeeping, filtering, etc. but just "What if I want to do something other than key word search?"
I find unexpectedly interesting articles all the time reading journals. I have yet to do so for a preprint server unless someone sends it my way, or it ends up on my Twitter feed.
Interesting. In math, I don't think anyone "reads journals", but lots of people check the arXiv frequently to see what has been posted.
Instead of deciding what gets published, peer review should help decide what gets noticed.
Are you going to delay publication of results from the LHC or JWST until someone has built another one?
The traditional unit of replication is a published paper, not a submitted paper.
A peer review takes positions on these things:
- does the paper correctly describe the state of the art, with appropriate references?
- does the paper claim and explain a novel contribution?
- is the methodology sound?
- do the claimed results justify the claimed contribution?
- does the paper clearly communicate to its audience?
That's it.
Replication will be attempted later if the audience cares about the results.
Nowadays a journal provides essentially no distribution, but there's no good alternative to journals as a "stamp of quality".
This should be considered a flaw (bug in tech terms) not a feature. why would you take decisions about funding something or not if you are not an expert. At least you are not expert in the same field but have knowledge and it is easier for you to communicate and discuss the proposals. But getting someone who never did a real scientific research and the last time he wrote a scientific essay was when he was in college to determine which research should be funded is wrong.
Meanwhile, I've been asked to evaluate hundreds of other mathematicians, on hiring committees and grant panels. Most of the applications are in specialities bordering mine, but not squarely in my area of expertise. Usually I can get a sense of what they've accomplished, but in most cases I can't reliably evaluate its depth or importance.
Unfortunately, external markers of quality are needed.
Because funding requires taking from something else -- either raising more revenue or reducing funding for any of the infinite possibilities that it could be spent on. And nobody is an expert in everything.
Where a particular area of expertise does cover the choices to direct funding, it generally does. With a lot of inefficiency and corruption and incompetence thrown in of course, but politicians and bureaucrats sure didn't decide how every dollar on the JWST would be spent.
Where is this the case? Federally-funded research generally involves experts. When Congress ascribes budgets to research agencies, those don't come from a vacuum. There are civil servants whose jobs revolves around providing expertise to budgetary oversight. They usually have titles like "program manager."
It seems to me that a return to this standard is inevitable:
* Scientific papers are in enormously difficult to peer review properly because fields have become so specialized
* The incentive structure is highly unbalanced, and the truth is, the people you’d like to peer review your papers are already too busy
* Peer review is very expensive (in terms of time) for a conscientious reviewer
* Many of these papers are low quality, serving to buttress the reputation of the author more than moving the field forward
Since it’s an open secret that so many journals are worthless, it would probably be better just to return to the publish fast model instead of trying to fix a sick endeavor.
This is assumed to be a bug, and not a feature. A lot of this comes from different perspectives, weighting, etc. Widening the number of reviewers isn't going to reduce the variance on this. You see it in grant review panels as well, which are much more "discuss and engage with your fellow reviewers".
"We live in a very different world today than we did when journals were circulated through print: One with the Internet, where data and research don’t need to be circulated in limited and expensive publications. Research can be shared almost immediately with massive networks of scientists through blogs, preprints [1] , data storage platforms and online forums, at little to no cost."
One of the assumptions here is that this access is uniform. It isn't. During COVID-19, some senior scientists essentially went with "Publication via press release", when more junior scientists can't do that. The popularity of figures on my field's Twitter hashtag is astonishingly non-uniform, cuts out whole swathes of people (govt. scientists), and is only lightly correlated IMO with the impact of their work.
I will say that, when I was a junior scientist, getting some of my work in good journals carried it much further than internet distribution would have.
"Finally, other types of centralized platforms could be hugely important too. Journals currently have exclusivity agreements (meaning researchers can only submit to one journal at a time) but they also have different formatting requirements, which take a long time to fulfill. By one survey estimate, articles are submitted an average of two times before they are published, and formatting takes up a total of 14 hours per article."
The number of journals that only require formatting post-acceptance is growing, and is awesome.
When it comes down to it, I think peer review's biggest strength is actually something the author takes as a negative. It's not "This paper has only been reviewed by two people", it's that every paper in this journal has been reviewed by at least two people, plus the public. For a lot of papers, I'm actually skeptical that that number is lower than if we went with "Well, the community will review it in the fullness of time".
Does anyone know of a mobile browser friendly way to read this kind of site without being constantly bombarded? It's exhausting. It seems like the sort of problem that would have been solved, but my attempts to discover the solution have failed.
I only had de reject all cookies prompt.
With firefox you can aldo use the reade mode, that extracts the main taxt and displays it plainly.
1. cookies 2. email newsletter 3. cookies again
The constant bombardment was more in reference to the state of the web generally. Popups and overlays that aren't a result of an interaction I've chosen to make are exhausting for me (and people with ADHD generally, and many others). I don't go back to sites that have them, but would be happy to read (and often pay for) the content if that wasn't the context.