Peer Community in X: A free recommendation process of scientific papers | Hacker News Reader

Peer Community in X: A free recommendation process of scientific papers

peercommunityin.org

74 pointseternalban18 comments

One of the problems with the current peer review system is the ease with which a reviewer can recommend rejecting a manuscript based on personal conflicts of interest (for example, viewing the work as competitive with their own). Making reviews open, and possibly de-anonymizing them, could alleviate this (allowing the public to review the review itself). Since the review requires the manuscript itself as context, the manuscript then needs to become publicly visible regardless of the reviewers' decisions.

Another problem is the difficulty that editors have in finding reviewers (to work for free essentially). One solution could be to require authors to act as reviewers before they can submit their own work. For example, if a manuscript requires N favorable reviews to be accepted, then for each manuscript an author submits, they must provide N reviews of other manuscripts.

There is still something missing from the above in terms of "reviewing a review". The closest thing I can think of is comment threads where the post is the manuscript... There needs to be a way for low-quality reviews to be reported as such and invalidated. Right now all this is on the shoulders of the editor, who doesn't have time to do more than add up the recommendations of the reviewers.

> to work for free essentially

I don't understand this. If you are an academic, or in industry R&D, then it's part of your job to take part in the community - you are being paid for it.

You pay to publish in a journal, you pay to have access to the papers of the journal and you review for the journal for free. Do you see the problem? You review for free but pay to access the content and to publish.

Note: the journal is not related with the people who pay you to do research.

People review my papers for me and don't ask for any extra money for it. So I review their papers for them and don't ask for any extra money for it.

Neither me nor my employer have never had to pay to publish any of my papers, but my employer does pay a few dollars a year for access to a papers repository run by a non-profit who just help the community come together - web hosting isn't free. I imagine it's one of the cheaper of the web services we pay for and a trivial cost of running a business.

I really don't see any problem.

Publish in a Congress is like $1000 (fee+trip), journals is less, access to the papers is millions for the universities.

They don't review the papers for you, they review for the Congress or journal, so they can decide which papers are worth to publish because otherwise they don't know.

Web hosting for universities would is nothing. So that is no an excuse to charge you for accessing the papers. Notice that the researches cannot have their papers in their web site because once they publish the papers the publisher has the rights.

Some things are changing but it is still so broken.

ArXiv-overlay journals like Quantum seem like a strictly better approach.

http://quantum-journal.org

Stated end goal of Quantum:

"The point of “becoming an attractive journal for researchers in every stage of their career” is crucial: Quantum should become an attractive publishing venue for the whole community of researchers in the field, including those who support Quantum’s vision but who, due to external constraints, cannot afford to take the risk of not publishing in a high-impact journal. Arguably this can be achieved in one of two ways: either the whole incentive system is overturned, or Quantum must become comparable to high-impact journals for the purposes of career advancement. In the short and medium term, only one of these options seems realistic. The decision to create Quantum as a selective journal is to a good extent based on the belief that it will give Quantum a much higher chance of making a difference." [source: http://quantum-journal.org/should-quantum-be-selective/]

I think it's an apples and oranges case given one (PCiX's) is a platform and the other (Q's) is a journal. It also seems to me that PCiX's goals are explicitly aligned with facilitating the sharing of results in a trust/reputation network, whereas Quantum seeks to address additional e.g career issues as well (which can limit it to more conservative decisions.)

As plainly written, that's a case of Quantum actually trying to consider real-world constraints, not a different end goal. It demonstrates seriousness.

Interesting. Romain Brette has been doing something similar by himself for computational neuroscience (and other subjects that catch his interest) http://journal.romainbrette.fr/

I like the idea, but I am afraid that as long as there are no "big names" behind it, it isn't going to make much headway in the conservative world of science. As tough as it sounds: reputation is the currency of science, and PCiX needs to get some before it can become a viable publishing alternative to the established journals.

Best scientific initiative of the entire history.

Just saying it because I had this idea 4 months ago (not that I'm claiming ownership of the idea, since it's a pretty simple idea that probably was thought of by thousands) and presented it to a friend which is a scientist (I'm not, thank you, God) and he promptly dismissed it.

Is there a reason to think this would be better than just computing the pagerank of every scientific paper based on the citation networks? (Optionally, use signals to predict future pagerank of recently published papers.)

I would expect that using the citation data from published papers would be more valuable, in the same way that revealed preferences are more valuable than polls due to the cost.

While Google Scholar implicitly uses pagerank to rank academic papers, I've put together a tool that explicitly shows you pagerank for each paper, available at [1]. Currently, this contains only pubmed, which means that it does leave out a significant set of physics/computer science papers, and also doesn't yet capture preprint archives.

1 = https://pubrank.carbocation.com/

To discuss how a paper is wrong you have to first cite the paper. The need to discuss bad papers can arise for a number of reasons. So some papers cite rank != quality.

True. Pagerank is "importance" rather than "quality". But as your network of recommenders becomes larger and starts to approximate the scientific community as a whole, I'd imagine that there will be convergence. If not, that would be a welcome and (to me) surprising result.

Curious. Just looked up my papers, most published in the last few years. How does the pubrank take in account time, or does it?

Pubrank currently is just pagerank based on the citation network. The common way to take time into account is to compute, for example, a "pagerank at 2 years". There are other ways to adjust for time (from linear modeling to more complex approaches) that I'd like to explore.

On average, for reasons that are not clear to me but which emerge from the data, pagerank seems to stabilize around 8 years after publication, for most publications.

I expected other people to suggest here, or maybe come up with a link of, a system totally decentralized of peer-reviews.

I really, really hope this takes off. Current system is so flawed...

Reply on news.ycombinator.com