Comment by mjn | Hacker News Reader

Comment by mjn | Hacker News Reader

I'm missing the leap here:

> The taxpayer ends up paying twice for the same research: once to fund it and a second time to read it. The heart of the problem lies in the reputation system, which encourages scientists to put their work behind paywalls. The way out of this mess is to build new reputation metrics.

The heart of the problem simply lies in closed-access journals, and a simpler solution is to move to open-access journals, which is already happening. Reputation is not an insurmountable barrier to this. In my field (artificial intelligence) the top two journals are now open-access. No fancy new metrics or removal of the concept of journal needed; problem solved.

I view the solutions proposed here as a bit more problematic. Download counts in particular are likely to just reward linkbaity research, rather than quality research, and exacerbate the already problematic race to put out misleading hyped-up press releases. Citation counts are liable to gaming as well, and are commonly gamed by both citation rings, and by people who consciously choose the subjects they publish in an ADDish citation-maximizing way, jumping in and out of hot areas to drop off a paper that'll turn up in searches. When it comes to judging the quality of AI research, I have a lot more trust in the editorial process of the open-access journals like JMLR or JAIR than I do in gimmicky new metrics that try to reconceptualize scientific publishing as gamification, with all the baggage gamification brings (mostly notably that it's not openly open to, but positively encourages treating it as a points-grabbing system to be gamed).

I don't have much trust in the supposedly "open" motives of this new batch of para-academic for-profit companies, either. Notice how academia.edu won't even let you download the PDFs of articles without registering for an account. I would guess the real purpose of these metrics is to set themselves up as new scientific gatekeepers in one form or another: to transition from a journal-oriented publication system to an "academic marketplace" oriented system where they own the marketplace. I have a lot more trust in the by-scientists, for-scientists model of JMLR.

Open access journals, based on authors paying $1,000-$3,000 to publish a paper, are going to be a temporary stepping stone on the way to the decline of the journal.

We need new reputation metrics to encourage the sharing of data-sets, videos, code, and generally the full range of a scientist's output. 75% of the world's scientific data doesn't get shared because the rewards aren't there for the scientist to share it. Journals aren't going to start publishing data-sets (or code, or videos, or blog posts), and so we need new reputation metrics, and reward systems, to encourage scientists to share this kind of information.

The other problem with the journal system is that the peer review system is slow and surfaces the opinions of only two or three scientists about a given paper. If you have a community of 20,000 people in the DNA sequencing research community, asking two people what they think about a paper is not going to deliver a statistically significant result. Furthermore the typical time-lag for the system to surface those opinions is 12 months.

We need a more robust peer review system which surfaces opinions from the entire scientific community in real time. Reputation metrics are the key to building that system. We need to provide the rewards for scientists to share their comments on papers, so they can build their reputations off the insights they share on each other's papers.

Yes, I get that's the sales pitch, but you didn't address gamification at all, which is what you're promoting, and will positively harm science, not improve it.

In addition, peer-review at well-run open access journals, which are the model I think we should be moving towards, does not take anywhere near 12 months, and in CS there are no publication fees (I gather this differs in biology). I typically get very well-thought-out reviews which help to improve my papers in about 6-8 weeks, and if accepted, publication is completely free of charge. And, it's becoming increasingly common to circulate preprints ahead of submission, on places such as the arXiv or at conferences or workshops, to get the feedback of a larger proportion of the community before publishing the final version of the paper.

I agree peer review does not settle the correctness of a paper; it's a system for ensuring a high-quality discourse, not to publish only papers which are the last word on a subject. That's what reply papers, replication studies, etc. are for. I am skeptical that attaching a bunch of points and badges to the discourse is going to improve the quality of that discourse, and suspect it will harm it. I'd rather look at how to improve the discourse by improving the journals. But that won't make money for VCs, so I'm not surprised that we're only seeing it from nonprofit organizations like JAIR and PLoS.

With the time-lag, you have to take into account the rejection rate. Most articles don't get into their first choice journal, and perhaps get into their second or third choice. You can't parallel submit, meaning that the time-lag between finishing the paper, and it getting published in a journal, often ranges between 6 and 18 months.

I understand that Computer Science does have a slightly faster peer review process than other fields. Here is a piece where a leading neuroscientist writes "Today the lag [in my field] between submission and acceptance is often more than a year". http://www.pnas.org/content/early/2013/01/25/201300924.short...

Gamification is an important issue. You should note that the journal's peer review system, based on two people reading a paper, gets gamed. There is the wide-spread practice of defensive citation, where you defensively cite anyone who might conceivably peer review your paper, in order to maximize the chances of its getting accepted. Some people know how to play that game better than others.

One way of handling gamification is to look at the quality of the people interacting with a piece of content, and not just the quantity. This is how Google's PageRank algorithm deals with the issue.

There is also the approach, taken by Twitter and Facebook, of picking out the people's whose judgements you trust, and then letting them act as quality filters. Those systems can be gamed too, but aren't typically gamed to a large extent.

I think you are confusing gamification and gaming the system, those are not the same thing.

GP was referring to badges, points etc (gamification) and you are referring to gaming the system (ie citing defensively).

If so, then mjn needs to clearly distinguish between (bad) gamification and a (good) reputation system, which will necessarily involve something like points or certifications.

"Defensive" citations are arguably consistent with good research practice in that they reference all relevant research by the most prominent names in the field; it's certainly no worse than the sort of open trading of citations a PageRank style algorithm theoretically encourages.

Sorry, I don't think Twitter and Facebook's quality filters "aren't typically gamed to a large extent" and I do think they're incomparably worse.

> Open access journals, based on authors paying $1,000-$3,000 to publish a paper, are going to be a temporary stepping stone on the way to the decline of the journal.

At least in my field, papers cost roughly $50k-$100k for just the researcher salary and overhead. Paying $2k to get a paper published is not a high hurdle (especially once the norm of closed-access publishing fades away). And this really seems like the minimum cost that needs to be incurred in order to get decent copyediting and the editorial cost of co-ordinating the referees. I love the ArXiv, but if I can get a the copyedited version from the journal I will.

Yes the referees aren't paid, which may or may not change, but paying a PhD editor $500 to figure out the best referees and engage in the back-and-forth with them and the authors is pretty minimal. So $2k for a professionally edited, polished paper which is then free to everyone forever seems like a pretty good deal, and essentially a non-issue.

Your point about the need for incentives for scientists to share non-paper scientific output like code is well-taken (and well-known). But I don't see why this can't be solved by open-access journals. I am completely unconvinced that facebook likes controlled by a for-profit organization are the way to go.

EDIT: By the way, you've mentioned in several places the 12-month time lag for publishing. This was solved for the physics community by the ArXiv; the papers get immediately released, and you can still submit to journals for the benefits of copyediting and refereeing. (Afterwards the ArXiv version is updated.) I don't know of any reason it can't work for all of academia.

You're right that the cost isn't the main issue. It shouldn't be disregarded though: with 2 million articles published a year, a $2,000 cost per article means $4 billion a year that could be spent on more research. To put that in context, the NIH's budget for tackling cancer is $4.9 billion a year.

The main issues with the journal system are the effects it has on the speed and quality of scientific communication. Peer review by two people is not robust, and it is very slow; secondly, the full range of scientific media aren't shared.

These are the problems that, I believe, require an overhaul of the reputation system in science. The only way to incentivize scientists to share their comments on papers quickly, and at scale, is to provide rewards for them to do that.

The range of media that scientists will start to share will be shared on platforms designed for that media type. Code will be shared on Github. Videos will be shared on YouTube. Data-sets will be shared on sites like Figshare or other sites dedicated to handling that content format well.

Reply on news.ycombinator.com