Well, however, we can argue here that a pre-print is not of the same quality as a published work. Not that published works cannot have errors (and be retracted), but at least they have passed a first-level scrutiny of an editorial board.
Machine learning is a field that's moving quickly and can also largely be checked quickly by other people in the field too. The problems that can't be checked easily are also those that won't be checked by the journals anyway (no journal I know of will retrain one of googles deep nets to see if they get the same result before publishing).
And please, I see what's happening here: I have been portrayed as the "traditional journal apologist" although it's definitely what I wanted here. I simply wanted to express my doubts that pre-printing is the answer to it all.
Anyway, is there a study that shows how many pre-printed articles in ML have been retracted or refuted so far? Until this is done, and shown as small as in peer-reviewed journals, keep a small basket.
I understand the process fully. It doesn't make it fast, nor does it make it a necessary cost to pay. Delaying access to content for several years does not solve a problem. A not-insignificant time was spent bouncing between people to sort out who was paying for the costs, then there is also a delay between acceptance and publication. This now averages just a month in pubmed, but papers can bounce around this point for a lot longer.
I am not arguing for pre-prints to replace traditional publishing, but the speed of spreading information is undeniably faster, and that's what started this whole chain of comments.
> Anyway, is there a study that shows how many pre-printed articles in ML have been retracted or refuted so far? Until this is done, and shown as small as in peer-reviewed journals, keep a small basket.
I don't get the phrase "keep a small basket", but no I've not seen this. The point is simply that PEERS (if we need to shout the word) can in many cases replicate the work and assess the results much more quickly than the traditional review & edit process. I picked the field because I see people re-implement work described in the papers very quickly, or the original authors share the trained models and code.
I'd also caution against using "retracted" papers as a measure, some journals charge for a retraction.
ArXiv is working very well for a lot of scientists in Machine Learning, since must results can be doublechecked by simply running the code. I won't disregard a proved approach that I've myself seen working just because it hasn't been published yet.
Could you provide some examples for a layman? To me it always seems that science moves far slower than one could hope for (e.g.: Every 'breakthrough' in batteries/Graphene of the last ten years and still no products)
Are you being literal about that or is that just a figure of speech?
Could you please provide a source for those claims, if true?
[0] http://journals.plos.org/plosmedicine/article?id=10.1371/jou...
[1] http://www.nature.com/news/1-500-scientists-lift-the-lid-on-...
Medicine on the other hand has the problem that you can't really afford large enough sample sizes to have sufficient statistical power.
This really isn't the major problem w/ modern medical research. In fact, if they had properly powered studies there would be far too many "discoveries" and the real problem would become obvious.
The real issue is that the efforts to come up with and study models capable of precise predictions (eg Armitage-Doll, SIR, Frank-Starling, Hodgekin-Huxly) have been all but choked out in favor of people testing the vague hypothesis "there is a correlation". There is always some effect/correlation in systems like the human body, so it is only a matter of sample size. As explained long ago by Paul Meehl, this is a 180 degree about-face from what was previously called the scientific method: http://www.fisme.science.uu.nl/staff/christianb/downloads/me...
Peer-reviewing has a lot of weak points, but saying that pre-print is the answer to them all is plainly wrong.
It's merely a supplement to peer-reviewed journals that has some nice characteristics, for some use-cases, which has been beneficial, to some researchers, in some fields.
What does "wrong" mean in this context? Does it mean that most scientific papers will be shown to be inaccurate in the due course of time? That is bound to happen given the nature of scientific progress.
But peer review aims to assess the methodology and rigour of the papers being presented. I agree that it is often debatable whether this happens in reality, but that doesn't explain your statement I quoted above.