Distributed Representations of Sentences and Documents (2014) [pdf]
arxiv.org
arxiv.org
He was merely advising the first author, who actually wrote the code. Source: (requires Google login) https://groups.google.com/forum/#!topic/word2vec-toolkit/XC7... and https://groups.google.com/forum/#!msg/word2vec-toolkit/Q49FI....
This highlights something people on HN don't appreciate about machine learning: how hard it is to actually trust results, and how likely it is that the results were affected by bugs in the code or how the dataset was handled. In this case the second author was only able to replicate if he didn't shuffle the dataset. Graduate students almost never write tests for their code.
Document Embedding with Paragraph Vectors (2015) https://arxiv.org/pdf/1507.07998.pdf
there is a subsequent paper by Mikolov and Mesnil at ICLR with the correct results. And code to replicate them.
Retraction? You want to retract a great paper because there is a 1-3% accuracy discrepancy for a result on a more or less random text classification task?
It's also disingenuous to trivialize the difference as 1-3% - that difference is more than a third of the actual error rate (7.42% vs. 11.27%) assuming you're referring to [1]. True, the IMDB dataset isn't a very important one, but it's important to clarify when you've made a mistake, especially if your paper has been cited hundreds of times.
I find the opposite true: code is easy to replicate and the datasets for algorithm comparison are open (e.g. imdb used in the PV paper). If you show very good results (especially with a simple approach such as PVs) people will immediately implement your algorithm and if their results don't match your published results, it will be known. PS: I implemented PVs shortly after it was published - though I don't care so much for the 1-3% or wtv accuracy discrepancy on the imdb dataset, the idea is great.
> Graduate students almost never write tests for their code
1) I doubt a standard software test would've helped here (probably cross-val would've caught it); 2) Who writes tests for experiment code? 3) The graduate student story is concerning: either a) someone doing a lot of the heavy lifting for the paper w/o being credited or b) this someone doesn't exist
In fact, dismissing a 3% difference is actually reflective again of how delicate understanding ML results is. A jump from 90% accuracy to 93% accuracy is massively different than a jump from 50 to 53% or even a jump from 80 to 83%.
Almost nobody writes tests for experiment code. You're proving my point :)
No. Graduate ML students can implement the papers they read w/o a reference implementation - just search github. As I said, I implemented PV w/o the reference code. Many others did the same even before I did.
> dismissing a 3% difference is actually reflective again of how delicate understanding ML results
Not really. I understand very well results in ML (Otherwise I would be a pretty incompetent graduate student). But does a 3% increase on say imdb translate to an increase on a another text classification task? possibly - but usually not. If it does translate well across text classification datasets, you will almost certainly see the different datasets and the results in the paper.
> Almost nobody writes tests for experiment code. You're proving my point :)
It's a good point but in my experience, the kinds of mistakes that I've usually found with my own or others experimental code would not be possible to catch with a software test. Only with analysis of the results do they become obvious.
Distributed Representations (1986) http://stanford.edu/~jlmcc/papers/PDP/Chapter3.pdf
"Each entity is represented by a pattern of activity distributed over many computing elements, and each computing element is involved in representing many different entities."
Full Book: http://stanford.edu/~jlmcc/papers/PDP/