A Worrying Analysis of Recent Neural Recommendation Approaches
arxiv.org
arxiv.org
Does this mean that the published results are garbage? No! But what does it mean?
First, don't miss the main claim: less than half the published papers have sufficient code and data available for replication. While one might argue exactly what the standards for availability should be, if journals and reviewers were to demand it, this number can definitely be improved. Greater availability of data and code makes for better science.
Second, my guess is that it means that paper authors are using excess "puffery" in an attempt to improve the chances that their papers are accepted for publication. They've decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement. Unfortunately, they are probably right.
The useful change (in my opinion) would be to change the standards by which papers are judged to be publishable. Reviewers should push back against weak baselines, but be more accepting of results that make modest claims. You can still have a useful paper even if the degree of improvement is small --- or even nonexistent. Maybe it will inspire someone else, maybe it's useful in an ensemble, or maybe it applies to a case where the baseline wouldn't. It's the puffery that's the problem, not the publication. Creating a system that rewards authors who more honestly evaluate their work would be an overall win.
I fear this contradicts your assertion that published results are not garbage. If researchers are claiming their systems are strong based on the fact that they beat weak baselines, then their claims are rubbish and their scholarship shoddy. If, on top of that, their work is difficult to reproduce, or replicate (by the distinction in the paper) then the published results are beyond garbage- they are meaningles garbage, garbage that fails to prove even its weak and insignificant claim to beat a weak baseline.
Basically, this is exactly the kind of shoddy scholarship that Backus accused social scientists of, in his infamous speech about "Cargo-Cult Science". In that speech, he points out that basing new research on previous weak research can only lead in the proliferation of weak results (my interpretation of his words, from memory).
This seems very much to be the case in machine learning research today. This article only provides further evidence that there is something rotten to the core in machine learning research.
>> Reviewers should push back against weak baselines, but be more accepting of results that make modest claims.
Agreed about the second part, however, the paper in the OP says that the way things are, it is very difficult to know which baselines are strong. My interpretation: because they're all trying to beat weak baselines that became established by beating earlier weak baselines. It's weak baselines all the way down.
Also, sorry HN, for the controversial comment (judging by the rollercoaster vote activity). I hope it adds something to the discussion despite the tone and I'll remember to keep the tone down in the future (but I keep saying that and I always end up giving in to the temptation to use more colourful language. I suck and I'm sorry).
It makes pulling out the actual content of the papers such a slog.
[0] exhibit A: https://ieeexplore.ieee.org/document/1017616
Source: am former producer/publisher of abomination research code.
Months of tweaking code is exactly what you want some test coverage for - so that you end up with a tweaked version of something where you still know what it does instead of something that does god knows what.
Edit: Perhaps one way to encourage this is via topic "reuse" papers, which present further detail on a previously published papers topic, but this time with code and more detailed discussion. It would at least help the situations in that the author gets some reuse of effort, but in a way that still shows new information and advances the field.
Literate programming suffers from this in general because you don't want to clutter up your document with noise about versions and so forth - even as appendices, including package information and build instructions is a lot of noise.
Requiring that papers making claims about some code base publish that code base simultaneously with the paper via the peer review process should be sufficient to improve the overall state of affairs.
Don't underestimate the value of demos and competitions with companion papers either, those tend to get a lot more notice.
In general, you'll have a lot of approaches that don't work out which are nice to have a record of, but definitely don't merit being in the final paper.
If your claims don't depend on specific behaviours of some body of code, you wouldn't need it - eg, an article claiming some asymptotic performance of an algorithm should describe the algorithm in the abstract s.t. the performance bound can be proven, not some specific language's implementation of the algorithm.
distill.pub has a journal that strongly encourages interactive articles: https://distill.pub/journal/
It's a lot of work, tho:
> One good test for whether your article is a fit for Distill is whether your collaborators and you are willing to put in whatever time is necessary to write and illustrate an outstanding article. In our experience, this often takes 100+ hours.
But for most journals, it's the story as usualy. Personally I dont feel very confident about a paper result if they don't publish the code. Why hold back on the code?
I know you know this, but this is absolutely a feature.
http://matt.might.net/articles/crapl
> The CRAPL is an open source "license" for academics that encourages code-sharing, regardless of how much how much Red Bull and coffee went into its production.
The big question is why this failure to publish code is tolerated by program+steering committees.
Besides, _ideally_, machine learning reseach is not about building a system but about making a theoretical claim that is supported by empirical results. The main claim should be about a property of the algorithm: its time complexity, its convergence, its correctness, etc. Experiments should directly test the theoretical claim and verify it in practice.
... ideally. In practice most machine learning papers are vehicles for a little table that compares your system with a baseline that you chose based on the fact that your system beats it (or you wouldn't have put it in the paper). Whether your experiments prove anything beyond the fact that you beat some benchmark is not really important.
And so whether someone publishes a reproducible paper is of no consequence because the only point of papers is to pass review and convince someone that the researchers should stay in the (research) money. In any case, things move so fast that most of the time nobody wants to reproduce your old and unsexy paper from last week.
Then some busybody comes along and points out the fact that you're wearing no clothes. Like in the paper above.
Also, more likely and non- nefarious, but no less harmful, there's just an academic culture that says "this is what a paper looks like and this is what it contains". Moving from this pattern increases the odds you'll not be published, and it would also likely result in a scandal when someone uses it to point out that the peers in peer review are neither peers (they can't read nor understand nor test the code) nor reviewing (they won't actually put resources into code review).
As a non-wealthy, non-academic, this isn't a battle I'm interested in fighting directly. There are other hills I'd rather die on :)
In some cases I basically had to run a test harness and brute force the parameters they'd used, because the algorithm was basically useless without picking parameters in the right range.
It was hard to accept then (~2008-ish), but now we're at a point where frankly at least for the more basic CS papers we ought to be able to expect source and some runnable format (e.g. a VM image) with test data.
I handed in a DVD with gigabytes of test data, full source, built binaries, and a test harness for the work I did for my thesis. All it took was to follow practices that I needed to follow to satisfy myself that my results were valid anyway.
I realize that for some work that relies on much larger and/or proprietary datasets making everything available would be hard, but a small test case illustrating the code run with the right parameters etc. ought to be a minimum unless there are substantial barriers, in which case the paper ought to signpost clearly what is missing and why.
The paragraphs of English text help them to be able to claim that they are indeed the authors of some novel method (exactly what the author wants).
Code/data accessibility helps others to both find errors/imperfections in their approach and to build upon their results (exactly what the author doesn't want).
I'm out in the trenches trying to turn ding dong papers into tools that does something useful, and the "muh mape 50% better" papers are effectively always something that should be ignored. Actually better algorithms and implementations (something like gradient boost machines/xgboost) are rare enough attempting to find them in the literature is futile.
Lack of availability of data and code should be an immediate disqualifier from publication for research. In any field. There is no substantial difference between withholding source code or data and withholding experimental procedures or list of materials tested. I can only think that the people who withhold such things are either actively malicious or are completely ignorant of the stupendous danger of cutting corners in science. Given the history we have to work with, it's pretty clear that the distance from cutting a corner in a way that seems harmless to the deaths of millions of people is very short. This is not something people should trifle with.
“What gets me is the thought that even trying our best to figure out what's
true, using our most sophisticated and rigorous mathematical tools: peer review,
and the standards of practice, we still get it wrong so often; so how frequently
do we delude ourselves when we're not using the scientific method? As flawed as
our science may be, it is far away more reliable than any other way of knowing
that we have.”
What do we have now:
1. We have a common platform, which is a de-facto standard for sharing (somewhat math-related) papers: arxiv.org. Basically, anyone can upload anything to arxiv.org, there's not much of quality review required. Which, all in all, I think is great. I mean, it's orders of magnitude better than other fields, where both publishing and accessing a published paper is expensive, painstakingly slow and inconvenient process which is obviously great for the journals that have a monopoly over it, but less so for everyone else. All being justified by the need of peer-review (which is not even sure to be working alright, because (1) we all know the quality of the vast number of studies in some social field is shit, and (2) you can even publish a completely bullshit paper in a very reputable journal if you try hard enough).
2. Much of the research is done in the academia (because, well, that's supposedly the reason it exists), but, paradoxically, the main moving force of the progress isn't really interested in progress. It is a competitive environment, so everybody is interested in publishing a paper which will be recognized as an "important one" later on, so the. They are not interested in anyone learning anything and especially not interested in making their rivals (and that is what any other researcher is, a rival) able to make a follow-up contribution easily.
3. Hence, many important studies by reputable scientists lack quality, which everyone knows, including the scientists. Papers are written in a passive-agressive tone, in particularly confusing terms an inconvenient format, they often lack the source code and the source data, a minor one-sentence contribution to the current state-of-the-art is spread around 20 pages which you have to read fully to understand what exactly the contribution is.
4. To summarize: everyone is interested in publicity of their work, nobody is interested in improving their work because it provides them the high ground, but everyone (including even those outside of the academia) is interested in improving others work, since it helps them to learn. Improvements on the others work, obviously, are not shared unless it can be framed as a follow-up study.
How (I think) it might be improved. Assume a platform everyone uses to share their work, let's call it arxiv2. arxiv2 has a ranking system, which pushes the papers it "thinks" are of the better quality to the top. "Quality" is (maybe not exhaustively, but that's the main idea) a set of domain-specific flags which are considered to be beneficial for a study to be conclusive in this field, like "has a source code", "has a source data", "has a benchmark/accuracy/specificity/precision metrics", "summary/abstract succinctly captures the main result". Depending on the field, features may be more complicated in order to address some common problems like p-hacking or non-representative sampling in the study. But all the metrics have to be concrete. Nobody is required to include anything, but if they don't, and if others mark the work as lacking some features, it has less chance to get to the top. Everyone can vote for the paper, some minimal moderation may take place. Disputes may be solved similarly to how they are solved on wikipedia/stackoverflow (which are, admittedly, awful and frustrating, but order of magnitudes better than the status-quo in the research).
https://arxiv.org/abs/1907.07355
"Probing Neural Network Comprehension of Natural Language Arguments
Timothy Niven, Hung-Yu Kao (Submitted on 17 Jul 2019)
We are surprised to find that BERT's peak performance of 77% on the Argument Reasoning Comprehension Task reaches just three points below the average untrained human baseline. However, we show that this result is entirely accounted for by exploitation of spurious statistical cues in the dataset. We analyze the nature of these cues and demonstrate that a range of models all exploit them. This analysis informs the construction of an adversarial dataset on which all models achieve random accuracy. Our adversarial dataset provides a more robust assessment of argument comprehension and should be adopted as the standard in future work."
In the field of computer vision, there are similar suspicions that state of the art computer vision models are overfit to ImageNet datasets and the like. The issue being that even average research labs do not have hundreds of thousands of dollars to reproduce extremely expensive and highly tuned models. Reproducing and advancing the field of deep learning is quickly becoming inaccessible to almost everyone except for a few of the highest funded industrial labs (Google, FB, OpenAI, Microsoft Research among a few others). This is not all negative, it's just that not everything can be taken as gospel.
I'm an outsider to NLP, but have been very surprised by the recent improvements, so would be interested in anything that covers their limitations and newer datasets which help us to get over them.
Have a look at the BERT & XLNet papers, they have results on a wide range of datasets, including SQuAD 2.0.
I think it's worth keeping in mind that all of the headlines about "surpases human performance" are 100% bullshit regardless of whether they are about vision, ASR or NLP. Performance on a dataset is not the same thing as real world usefulness.
But this takes nothing away from the fact that ELMo/BERT/XLNet make it far easier to pull a model off the shelf and fine tune it for your task and get good performance.
And the gains here are so broad and not task specific that issues with individual datasets are not really an issue, it is the massive gains in pretraining that are the real story, and if that even helps you overfit a test set, it shows that they now have enough built-in linguistic knowledge that we previously didn't have to even overfit a bad test set.
This is obviously bad news for you if you thought that these tasks were basically solved, but it's completely irrelevant if you were building systems since these methods all result in relative gains on whatever language task they were pointed at.
The bigger issue IMO with many recent NLP papers is not that they are expensive to train from scratch, which is a one time cost, but that they are expensive to even do inference with, which makes it hard to actually deploy them in real systems.
But if your question is "is recent progress in NLP illusory" I think the answer is clearly no since the gains were across a broad range of tasks, and frankly I've used BERT and it makes my application better, the main issue I have is the runtime issue.
Or in it, either. ML researchers are painfully aware of how the sausage gets made. (About 12 hours before the NIPS deadline, typically.)
[1] 1 quick citation before I get back to work: https://www.bewellbuzz.com/technology/many-published-studies...
It's terrible, but no more terrible than many other comparable fields.
My point is that the findings of this article are not necessarily indicative of machine learning being junk science, but it's more likely indicative of a systemic problem with all research, where there's incentive to publish the paper, but not the data, code, models, or even generally enough information of any kind to successfully replicate the experiment more often than 10-15% of the time.
I remember when LIGO first detected a black-hole collision, they released the raw data and the Jupyter notebooks that were actually used to go from the raw data to the published results. Everyone was floored. It shouldn't be that amazing - it should be the standard. For every field.
Ha. I get a feeling there's alotta overpaid data scientists out there.
Someone on the company board has just read a pop sci article about "machine learning" and now as a result the company has engaged some expensive consultants to advise us on "potential machine learning opportunities." The consultant spends maybe a day or two touring one of the company's plants and has only the most broad and high level view of how our industrial process actually works. The consultant then promises the world "machine learning is great, you can use it here, here and here for sure. It will save you millions..." It gets the board members excited.
So the company picks some toy problem and gives the consultant and his team a six month contract to deliver a proof of concept. The deadline passes a solution is delivered - it is underwhelming. The contract is not extended.
edit: All cynicism aside I appreciate the possibilities but for ML to succeed the developers really need to be embedded/integrated into the company have a good understanding of problem space and fundamentals. Not really something you can outsource easily and expect to succeed.
This struck me as completely terrifying... it's not like the set of HVAC units was changing out every week. And one would hope that the historical data would help arrive at a general solution. My guess is they had some sort of church-meets-state problem in the test/train split, so that the short-term data allowed 'predicting' the test set.
Dude was quite convinced that they were making HVAC history, though.
Even if these models are outperformed, would they be useful anyway, to serve as insights into the functioning of an agent that can deduce solutions to these tasks "from scratch?"
After all, the human brain is a wondrous AI, but it is often also outperformed by simple heuristics.
Which is probably your point :P
Right tool for the right job. The problem is that when you have a hammer called machine learning, every problem looks like a machine-learning-sized nail.
Facebook -> make the user see more ads
Amazon -> make the user buy more things
You probably want to optimise for keeping people on your service, first (and last).
For instance the NCF and MCRec papers tuned model parameters on the test set and the SpectralCF paper used a non-randomly sampled test set for evaluation.
That to me is even more surprising than their revelations that a well-tuned statistical baseline outperforms these models.
I think the greater issue is that researchers are afraid to admit that their results aren't great. And I don't blame them- it could impact funding, access to journals, etc. But mediocre results are still important for research, and it's a shame the scientific community doesn't recognize that.
No, being "understanding" about this makes you part of the problem.
Part of research is trying new things, and exploring, but you should always be checking this against simpler techniques and currently accepted best performance. If it isn't an improvement, you don't publish at all unless you have something interesting to say about why it isn't improving things. Or perhaps in a methods survey.
One thing that happens when a technique like deep learning gets popular and things like python toolkits pop up making it easy to try, is that you get researchers in adjacent areas who don't really understand what they are doing trying to apply it to their favorite problem domain, and then comparing almost exclusively to other naive groups doing the same thing. It can be hard to tell if there is anything interesting there, even when there are dozens of conference papers, etc.
Basically the same thing happened with kernel methods when SVMs were hot, to a smaller scale.
Compare this to say, AlexNet. The reason that was immediately obvious something interesting was happening there was the fact that lots of people who did know what they were doing in models, had tried lots of other approaches, and you could make direct comparisons.
So yes, blame them. I do think negative results should be valued higher, but the fact you did some work doesn't make it publishable.
Frame it another way, if you give me a paper proposing a complex model and I grab your data an play with it a bit and it turns out linear regression with appropriate pre treatment works better ... well then I'm forced to believe you are either incompetent or being dishonest. Or, if students, your supervisors are.
This generalizes well. You should always be comparing against a known (or considered) good solution on your data under the same conditions, not just comparing against last years conference paper and variants you are trying to "improve". The right choice of baseline comparison will depend a bit on the domain, but not including it at all is shockingly poor scholarship.
I've even seen paper submissions with no direct comparisons at all, because the "researchers" didn't have access to the comparators data, and were too lazy to implement the other methods. Which leads to another sloppiness - methods that get pulled into comparison not because they are the right choice, but because there is a publicly available implementation. In the best case this forms a useful baseline. In the worst case, well, I guess it's good for your citation count if you implemented it :)
I don't mean to suggest that papers should boil down to performance comparisons with baseline results, not at all.
What I'm saying is that if you don't do this comparison somewhere, it can be very hard to tell what your numeric results do mean. In the worse case you see people offering numerical comparisons to other approaches that are similarly unpinned, and you can't tell if they are interesting even if they are apples to apples.
As a researcher, you are being lax if you never do that work if for nothing else than a sanity check on your implementation. If you've done it, it's good to include in the paper as a point of reference. If you currently aren't performing better than baseline, that's fine - but you should understand why and discuss that with insight too.
So it isn't the focus. But it is table stakes that you understand this stuff.
Research (noun): "the systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions."
Q: How can one reach new conclusions without being aware of existing approaches and conclusions?
...so, we are in agreement then?
I would imagine most people in academia know that most papers published are published "just to publish" to help one's academic career, mostly by MSc and PhD students. They are essentially "write-only" papers. Everyone has to start somewhere.
Now, if we saw something like this out of DeepMind or OpenAI or in Nature, that would be worrying.
Conference | reproducibility | h5-index
----------------------------------------
KDD | 43% | 77
WWW | 33% | 70
SIGIR | 25% | 55
RecSys | 13% | 4018 papers/algorithms were tested
11 weren't reproducible
7 were repoducible with considerable effort but only 1 clearly outperformed the baseline but still didn't outperform a well-tuned linear ranking method
--
Travis Ebesu et al. 2018. Collaborative Memory Network for Recommendation Systems. In Proceedings SIGIR ’18. 515–524. was the only paper out 18 that was both reproducible and outperformed the baseline but did not consistently outperform a well-tuned non-neural linear ranking method
Reproducible with considerable effort but outperformed by comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques.
====================
They are:
kDD:
[17] Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S Yu. 2018. Leveraging meta- path based context for top-n recommendation with a neural co-attention model. In Proceedings KDD ’18. 1531–1540. [SIGIR]
[23] Xiaopeng Li and James She. 2017. Collaborative variational autoencoder for recommender systems. In Proceedings KDD ’17. 305–314.
[48] Hao Wang, Naiyan Wang, and Dit-Yan Yeung. 2015. Collaborative deep learning for recommender systems. In Proceedings KDD ’15. 1235–1244.
RecSys:
[53] LeiZheng,Chun-TaLu,FeiJiang,JiaweiZhang,andPhilipS.Yu.2018.Spectral Collaborative Filtering. In Proceedings RecSys ’18. 311–319.
WWW:
[14] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings WWW ’17. 173–182.
[24] Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018. Variational Autoencoders for Collaborative Filtering. In Proceedings WWW ’18. 689–698.
Non-reproducible:
====================
KDD:
[43] Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Multi-Pointer Co-Attention Networks for Recommendation. In Proceedings SIGKDD ’18. 2309–2318.
RecSys:
[41] Zhu Sun, Jie Yang, Jie Zhang, Alessandro Bozzon, Long-Kai Huang, and Chi Xu. 2018. Recurrent Knowledge Graph Embedding for Effective Recommendation. In Proceedings RecSys ’18. 297–305.
[6] Homanga Bharadhwaj, Homin Park, and Brian Y. Lim. 2018. RecGAN: Recurrent Generative Adversarial Networks for Recommendation Systems. In Proceedings RecSys ’18. 372–376.
[38] Noveen Sachdeva, Kartik Gupta, and Vikram Pudi. 2018. Attentive Neural Archi- tecture Incorporating Song Features for Music Recommendation. In Proceedings RecSys ’18. 417–421.
[44] Trinh Xuan Tuan and Tu Minh Phuong. 2017. 3D Convolutional Networks for Session-based Recommendation with Content Features. In Proceedings RecSys ’17. 138–146.
[21] Donghyun Kim, Chanyoung Park, Jinoh Oh, Sungyoung Lee, and Hwanjo Yu. 2016. Convolutional Matrix Factorization for Document Context-Aware Recom- mendation. In Proceedings RecSys ’16. 233–240.
[45] Flavian Vasile, Elena Smirnova, and Alexis Conneau. 2016. Meta-Prod2Vec: Prod- uct Embeddings Using Side-Information for Recommendation. In Proceedings RecSys ’16. 225–232.
SIGIR:
[32] Jarana Manotumruksa, Craig Macdonald, and Iadh Ounis. 2018. A Contextual Attention Recurrent Architecture for Context-Aware Venue Recommendation. In Proceedings SIGIR ’18. 555–564.
[7] Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat- Seng Chua. 2017. Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention. In Proceedings SIGIR ’17. 335–344.
WWW:
[42] Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Latent relational metric learn- ing via memory-based attention for collaborative ranking. In Proceedings WWW ’18. 729–739.
[11] Ali Mamdouh Elkahky, Yang Song, and Xiaodong He. 2015. A multi-view deep learning approach for cross domain user modeling in recommendation systems. In Proceedings WWW ’15. 278–288.