Hacker News Reader

Comment by mediocrejoker | Hacker News Reader

I think a lot non-academics (myself included) tend to hear the phrase "peer reviewed" and conflate it (rightly or wrongly) with "these results are replicable and therefore correct."

I don't know what the long term solution is but I have a feeling it won't involve companies that charge a lot of money for a service (publishing) that has been effectively free since the Internet has existed.

View on HN

A lot of peer reviews in ML are currently completely open. For instance you can view the reviews for papers in previous and upcoming conferences here [0].

Also in terms of replicability ML is in a unique position where a lot of research is on standard datasets with models that don't take an excessive amount of compute power to train. This means that if the authors include a github link to their code you can fully replicate their results. So some "peer reviewed" ML papers can actually subscribe to your first description of peer reviewed.

[0]: https://openreview.net/

I think a lot of ML research, despite being replicable is not necessary useful, because what we really want, approaches we can add in addition to all the other methods we have, that introduce meaningful improvements without unnecessary complexity and work across datasets, are quite rare. And that's assuming the evaluation was done well.

Not to say the research is worthless, just that straight replicability is not necessarily enough.

Reply on news.ycombinator.com