Hacker News Reader

Comment by jbob2000 | Hacker News Reader

jbob2000

edit: people don't like discussing things, just want to see their own viewpoints reinforced.

I work with a fair amount of people from academia, a lot of them seem to have the view that things that have not been peer reviewed are not worth reading.

But ML already has conferences and journals where work is peer reviewed and published in an open access manner.

So it's not that academics see journals/conferences as value-less, but that they are already doing all this work for free, why do it for free for Springer who will then charge everyone?

I think a lot non-academics (myself included) tend to hear the phrase "peer reviewed" and conflate it (rightly or wrongly) with "these results are replicable and therefore correct."

I don't know what the long term solution is but I have a feeling it won't involve companies that charge a lot of money for a service (publishing) that has been effectively free since the Internet has existed.

A lot of peer reviews in ML are currently completely open. For instance you can view the reviews for papers in previous and upcoming conferences here [0].

Also in terms of replicability ML is in a unique position where a lot of research is on standard datasets with models that don't take an excessive amount of compute power to train. This means that if the authors include a github link to their code you can fully replicate their results. So some "peer reviewed" ML papers can actually subscribe to your first description of peer reviewed.

[0]: https://openreview.net/

I think a lot of ML research, despite being replicable is not necessary useful, because what we really want, approaches we can add in addition to all the other methods we have, that introduce meaningful improvements without unnecessary complexity and work across datasets, are quite rare. And that's assuming the evaluation was done well.

Not to say the research is worthless, just that straight replicability is not necessarily enough.

The greatest value of journals isn't that three random people (give or take) have reviewed and suggested edits to the papers before publication.

The greatest value of journals is when hundreds or thousands (or, hell, dozens) of other people, well informed and competent in the subject matter, read the papers and consider how it meshes with their own experience and understanding and build upon it, writing letters to the journal pointing out problems and articles which follow up on the original work.

That is how an idea is validated. An article published in a journal that no one reads isn't much different than an unpublished article, and if work is spread, responded to, and built upon outside of journals, it isn't much different from a journal-published article.

That's unfortunate because that is how consensus and dogmatic group-think emerge. People need to think for themselves.

I love learning but stayed away from college because I couldn't stand how fundamentally at odds traditional education routes are to creativity and free thinking.

Personally, I find Twitter and arxiv-sanity good sources for keeping up with ML. They have their own mechanism of surfacing popular content and filtering based on preferences.

> a lot of them seem to have the view that things that have not been peer reviewed are not worth reading

Can't we have some kind of PageRank system, not necessarily based only on citations, to solve that?

Pagerank is a kind of citations system.

Yes, but I suppose you can also use it for upvotes instead of direct citations.

That role (in machine learning) is already filled by conference proceedings (plus ArXiV), so this journal is an opportunistic attempt to monetize a niche which doesn't need filling.

In some fields, like machine learning, many researchers put their work onto arxiv. There is like 50 papers a day now. https://arxiv.org/list/stat.ML/recent

People don't find out about research by reading journals. They go to lectures and read mailing lists for that. Journal articles are typically a year too late.

The value-add from journals is that they (at least theoretically) guarantee another person has read the paper carefully and believes it is correct.

With biology at least a lot of your reputation relies on your reputation, and a lot of that comes from publications and the impact factor of the journals you publish in, Nature being the one to aim for. Not sure if its as relevant to AI.

Computer science doesn't need journals because (1) It's generally easy enough to reproduce any work you care about, so you don't need to take anyone's word for its correctness, and (2) researches can make a fortune in industry doing their (profoitable) work or something related, so the don't need to suck up prestige factories in order to get funding.

I was thinking as well that its less abstract. Biology research takes a lot time and of work to filer through to practical applications. AI research is generally doing something practical straight away.

The old model may be obsolete, or at least need tuning. What about offering a "vetting grade"? Anybody can submit (assuming minimum standards) and anyone can read, but you'd have to pay to have your research vetted, perhaps with different levels of vetting. Don't make it all-or-nothing.

Reply on news.ycombinator.com