http://nyus.joshuawise.com/batchnorm.pdf
... which references an even better one:
http://pages.cs.wisc.edu/~kovar/hall.html
We've been having a solid laughfest in the office for the past 10 minutes or so.
This reminds me of my bioinformatics class. The final project was to reproduce the results of a famous paper in the field.
All of us spent _weeks_ trying to do it. Nobody succeeded. The more we dug into the paper, the more holes appeared. There were variables missing in the paper, assumptions not covered, datasets not properly specified, etc. It made reproduction nearly impossible; like winning the lottery. Imagine trying to recreate the results of a deep learning paper without the paper specifying _any_ information about the layers used, their sizes, or any hyperparameters.
The professor was equally mystified.
Years later I learned this kind of pseudo-science is rife in the field of bioinformatics. I felt both a sense of relief in knowing we weren't crazy, and disappointment. I actually really liked that class; the field of bioinformatics fascinated me. But realizing what a cesspool it was, left me disappointed.
I'm glad machine learning as a field has taken proactive steps to avoid these exact kinds of issues. It's now common practice in ML to publish code and models alongside your papers, and most ML libraries allow deterministic training. This makes reproduction of results easy. It's a breath of fresh air. That doesn't obviate all problems. Methodologies and conclusions are still up for debate in any given paper. But at least the experiments themselves are reproducible. And if you question the methodology or some aspect of the experiment, you can go in and augment the experiment yourself.
Dieselgate started with a team of students attempting to reproduce VW's claimed emission numbers.
This was a decade ago. Looking at the paper again I believe we only tried to reproduce a small portion of it; the phylogeny tree from the paper and its supplemental material.
Yeah about that, I've got some bad news...
A friend of mine spent a good chunk of his PhD trying to reproduce an experiment involving growing primary cells in serum-free medium (the idea was to use that experiment as a starting point, and explore more aspects of it). The protocol was:
1. Grow some regular immortal cells in serum-based media in a dish, so they coat the dish with extracellular matrix
2. Use trypsin to detach the cells from the dish and remove them
3. Wash the dish carefully to remove all traces of serum, but leaving the extracellular matrix
4. Plate the primary cells onto the dish and grow them in serum-free medium
He tried for months and couldn't get the cells to grow. Then he got sloppy, didn't wash the dishes as carefully as he should, and bingo, the primary cells grew fine, as described in the original paper.
After some subtle digging, the inescapable conclusion was that the original authors had not washed their plates all that carefully either, and the serum-free medium was not exactly that. The whole premise of the experiment was flawed.
Sounds like a site dedicate to "My Ass" results would be extremely popular with grad students and real world researchers. Being able to know "it's not just me" and maybe even avoid some of the stumbling blocks others have run into, or to not just blindly use some approach that happened to work for one experiment, but seems to fail for many others.
I agree in general, but I'd also love to see those published as actual beautiful papers, not just ugly formatted websites. (Okay, the website in case isn't that bad. At least it's clearly structured and readable.)
These would be mostly short papers, for sure. But there could be a separate section in the journals for them - just like the "outtakes" section at the end of a movie.
But if this encourages other people to write a follow-up paper that fixes the issue, it would still serve an important purpose.
Medicine has a good tradition of adverse clinical writeups. "Patient presented with X symptoms, I administered Y treatment as recommended by [Z], and the patient got worse." One such writeup isn't conclusive evidence against Y, but suggests an issue to look into.
> Following the popularity of MapReduce, a whole ecosystem of Apache Incubator Projects has emerged that all solve the same problem. Famous examples include Apache Hadoop, Apache Spark, Apache Pikachu, Apache Pig, German Spark and Apache Hive [1]
Looking at his resume, he did wisen up and did his master's thesis in computer science. I trust he's happier now than as a undergrad student.
The standard technique is to set up a "Kelvin probe", with four contacts on the Ge sample. Pass a current from a constant current source (an IC or FET these days) between the outer two contacts and measure the voltage across the inner ones.
It doesn't sound like his lab assistant set up something at which he could succeed, and that's a shame. He couldn't even repeat the room temperature reading.