Hacker News Reader

Comment by liliumregale | Hacker News Reader

Let's distinguish between papers and preprints, please. arXiv has contributed to a blurring of the distinction. The arXiv preprints are useful but should always be taken with a grain of salt. There is nearly no filtering done on things uploaded to arXiv.

Everyone accessing someone's uncritically reviewed work is a bittersweet gift.

View on HN

In mathematics, at least, papers and preprints are indeed widely considered to be the same thing. In practice, for people working in the field, they are.

Math papers tend to be highly technical, read by other specialists in the field. When it comes for correctness -- whether or not I should take a paper with a grain of salt -- the authors' reputation counts for much more than the journal's. And in case of student authors, who are just beginning to publish, the advisor is implicitly staking their reputation on the work as well.

There are also preprints on the arXiv, written by people unknown in the community, claiming to prove the Riemann Hypothesis or some such. These aren't taken seriously by anyone.

An outsider might not be able to tell which preprints can be considered equivalent to papers, but such people are not likely to be seriously reading math research in the first place.

You can always overlay a reputation system on top of your pre-print server.

The informal one you describe here, or any formal one you can come up with.

Arxiv has been working just fine for a long time, there's no need to change it. Besides I'm not going to voluntarily post my work so I can get publicly rated by a bunch of unknowns lol.

You're thinking of social-media-type "reputation".

Instead, think of the goal being to associate measures of worth with the reviewers. If you're publicly rated by a bunch of worthwhile people, count yourself lucky.

> Arxiv has been working just fine for a long time, there's no need to change it.

Exactly, that's why I am not suggesting any change to Arxiv.

Think more of people eg submitting Arxiv URLs to Hacker News for what I have in mind. Or discussing Arxiv submission on a forum or in a wiki etc. You can imagine some specialised software that has some better support specifically for material from Arxiv.

That's what I mean by 'overlay'.

Or think of Slatestarcodex publishing a blog post with links to his favourite Arxiv papers for that month. That's pretty much equivalent to what a journal produces. And if Slatestarcodex compiles that link list by doing some peer review and chatting with the authors of the papers, that's almost exactly what the journal does.

Yes. For example, here is a paper by some Cornell people where they reinvent machine learning model evaluation with the only motivation that I can tell is hubris and self service:

https://browse.arxiv.org/pdf/2310.02335.pdf

Do not trust arxiv papers. They have not been vetted.

> Everyone accessing someone's uncritically reviewed work is a bittersweet gift.

Review work is not always done by senior researcher (e.g., professors). Senior researchers often hand this down to PhDs. Having 3 to 4 reviews by nice junior reviewers doesn't sound very critical.

They have to say they did this and you are forgetting the editor's role in paper evaluation. This criticism can and is taken into account and you can send papers out for more reviews if you get conflicting ones. In my experience as an editor, junior people typically give better reviews than senior (unless they are emeritus and then have unlimited time). I suppose this has to do with confidence in the junior person who will question their review themselves.

Just to be clear: you'd expect PhD students to be trained in reviewing by their supervisors.

So PhD students writing the initial review is not weird - it is an expected part of their training. As is the supervisor going over the review and providing constructive feedback. As is the review being submitted under the supervisor's responsibility, with credits (mention in proceedings) to the student for acting as a subreviewer.

Yes, there are ways to abuse this system and yes, abuses do occur. Any system for gaining job prestige or workload reduction is a target for gaming. This doesn't mean the system should be thrashed, but it does warrant additions to curb excesses.

If a late-stage PhD student in the same narrow technical field can't review the paper, then it's almost certainly a problem with the paper. After all, junior people are the primary audience for any paper. Also, PhD students often have more depth on their research topic than the professors.

The sibling comments about making sure that most reviews are written by senior researchers also make good points. That should be checked by the program committee or editor.

Arxiv paper quality is better than journals' average paper's quality. Because publishing in Arxiv doesn't count as paper in resume in many places, there are far fewer papers who publish just for resume.

It’s how science worked for 3 centuries before the current review system was instituted just a generation ago.

Let's do a quick analogy. arxiv = github. It's all collaborative writing, right? You publish data, code, and your paper continuously. Then you have releases. Perhaps they get tagged with what publication venues accept them.

I'm confused. Do you accept published papers as gospel? They should be taken with a grain of salt too.

Depends on the field certainly. A paper in the Annals of Mathematics is definitely a lot more rock solid than whatever goes on the arXiv, or reviewed papers in certain fields that are particular magnets for junk science.

Funny you should mention Annals. A journal famous for publishing two papers in three years by the same author, one proving some theorem, and the other disproving the theorem. Sure, tons of other journals have done so, but Annals is definitely the highest profile one. Maybe take a look at https://mathoverflow.net/questions/282742/endless-controvers... or https://mathoverflow.net/questions/35468/widely-accepted-mat... It's also a nice way to pad your CV if you manage to get the wrong theorem published - you get two Annals papers for the price of one.

It is of course true that published papers have been vetted. But very often, it simply means that 1. an editor glanced at it, 2. (optional) a peer provided a positive quick opinion on the paper, without checking the correctness, 3. one or two independent referees presumably read the paper and produced a report on it. It's not nothing, but it doesn't mean you should accept blindly as truth everything published.

For context, I'm an associate professor of mathematics at a large research university.

The way I look at it, we passed the point where there are so many people that no one can read all the papers in their field any more.

Peer review is the first filter that papers go through. It's not perfect (it makes mistakes in both directions), but the output of the peer review process definitely has a higher signal to noise ratio than the input.

Reply on news.ycombinator.com