Hacker News Reader

Comment by MAXPOOL | Hacker News Reader

You would think that computer science can't have replication failures, but it can. And I'm talking about my own field: machine learning. There is so much hype that I suspect that people are pushing papers and trying to actively hide the irrelevance of the methods and algorithms they develop.

Artificial intelligence faces reproducibility crisis http://science.sciencemag.org/content/359/6377/725

Reproducibility in Machine Learning-Based Studies: An Example of Text Mining https://openreview.net/pdf?id=By4l2PbQ-

Missing data hinder replication of artificial intelligence studies http://www.sciencemag.org/news/2018/02/missing-data-hinder-r...

>In a survey of 400 artificial intelligence papers presented at major conferences, just 6% included code for the papers' algorithms. Some 30% included test data, whereas 54% included pseudocode, a limited summary of an algorithm.

View on HN

> 6% included code for the papers' algorithms

This is completely ridiculous.

In pure mathematics, everything you need is right there in the paper. If instead you're in the physical sciences, you obviously can't include your lab in the paper. For software, it's perfectly possible to include the lab in the paper, so to speak, and there's no excuse for depriving the rest of us of access.

Not interested in releasing the source? Ok, just don't brand your project as 'computer science'.

Including full source-code (and ideally data-sets, unless there's good reason this can't be done) should be a basic requirement for serious publications. It's disappointing that this isn't (yet?) the norm.

</rant>

Agreed, at least pseudo-code can be included.

I have to strongly disagree here. The source code itself - and not some proxy for it - must be made available.

Pseudocode is just another description of the approach the authors took. It doesn't allow another researcher to carefully recreate the exact experiment that the authors ran.

Software is famously hard to get right. This can be exacerbated by certain research problems where you don't know the result to expect (modelling, say). If you're going to publish results, the source you used should be available for inspection, for similar reasons to why mathematicians have to publish the proof, not just the conclusion plus a promise.

The upsides here are many: protecting science against software bugs, easier replication of experiments, better protection against academic fraud, and helping other researchers extend the work.

(I realise this last point may be at odds with toxic academic careerism. All the more reason for the publication process to insist on it.)

Most papers include pseudo-code when it helps clarify things. This tends to be a very high-level description of the algorithm though, and is not really the same as including actual source.

You're not alone.

https://www.artifact-eval.org/

I was talking to a friend who moved from the Google Brain team to the DeepMind team and he said point blank that no one was going to reproduce their work and that they were not going to reproduce other groups' work. Everyone has their own research agenda and the cost of the hardware and compute time needed to process the data is available to a tiny number of research groups around the world.

> the cost of the hardware and compute time needed to process the data is available to a tiny number of research groups

You'd think this would make them more interested in replication, since otherwise the likelihood that they are actually doing something of value is uncertain.

Science and research is full of all kinds of false starts, you don't increase your speed by blinding yourself.

Falsification is the very definition of science.

I do research in psychology and also a little in machine learning (classification using text samples). I've actually been working with the Many Labs data (the project that's the focus of the article).

My impressions of machine learning match up with yours. There's just so many parameters that there's so many opportunities to capitalize on chance, combined with [necessarily] huge datasets that preclude replication, that it's hard to avoid. I have also been surprised at how much tweaking there is of parameters; I'm used to work in traditional statistics where more is derived from theoretical principles, as opposed to trying a bunch of values to see what "works." This all lends itself to overfitting. I strongly believe that a lot of the adversarial input work is basically capitalizing on this overfitting.

To be fair, I don't think people are necessarily being nefarious, I think people across all fields just don't appreciate the dangers of overfitting.

The one upside of this Many Labs work, mentioned in the paper, is that it tends to show that the common criticism of "how can you generalize from this sample of undergrads" is not so much of a problem. Not that it's not an issue at all, but if you're studying some basic cognitive process it's probably not going to matter that much if you use undergrads versus some perfectly representative sample of the population. People have shown this in different ways before but it's useful to know more about. Obviously with some things sociogeographic variation will matter more though.

One thing that might not be totally obvious from this article and others is that although some effects clearly replicate, and others do not, there are some effects that seem to be in a grey area, where the effects are probably real but much smaller than originally reported. The distribution of effect size estimate distributions is more continuous, with shades of grey, than these news reports would have you think. Whether or not it matters that an effect is tiny versus zero might not practically matter, but at some level it is important to be mindful of.

> One thing that might not be totally obvious from this article and others is that although some effects clearly replicate, and others do not, there are some effects that seem to be in a grey area, where the effects are probably real but much smaller than originally reported.

The two prior replication studies that I know of - the first was several years ago and raised the issue in the public mind afaik, then second maybe in the last year - both replicated almost all of the effects. The problems were that some effects were weaker than in the original studies. Are you saying that this most recent attempt at replication shows a large number of studies with no significant effects at all?

real effects might be weak enough that they only make it through the first time (non replicative) publication process when chance makes the effect seem stronger than it actually is.

> real effects might be weak enough that they only make it through the first time (non replicative) publication process when chance makes the effect seem stronger than it actually is.

That's hypothetical. What actually happened was what I wrote in the GP: The effects were replicated almost every time, except sometimes they were weaker.

you miss my point. the strength of a novel main finding must be sufficient to warrant the opportunity cost of trying to get it published. a weak but real effect will be more or less strongly indicated in an experiment, just by the vagaries of the sample, but weak results typically will not get published. it is thus not surprising to me that findings might frequently be replicable but weaker than initially was found.

In my mind, 50% replication is not that horrible. Even decent seeming sample sizes might have an 80% chance of replicating at a certain effect size, and the effect size originally reported could be (by chance or other reasons) over-stated, leading to insufficient power. More certainty ends up meaning vast increases in sample size). Another issue that should be tested (have not read paper but doubt it) is not just whether the result is statistically significant, but whether it statistically differs from the original published result. If the effect is weaker, but not statistically different from the published result, I would hesitate to say the effect is not real. Then there is that some of the psychology findings investigated were rather dubious to begin with, and the ones that even non-specialists can identify as being likely to be true replicated.

So, a result that is statistically no more accurate than flipping a coin is acceptable to you? Large cultural shifts have been pushed on a large scale based on studies that may have no more probability of truth than flipping a coin and this is OK to build a society on? I couldn't disagree more.

Your characterization of what I said is overly simplistic and ..wrong. I am saying that even with a good sample, replication studies only have about at most an 80% chance to replicate a true effect for studies that had reported a "moderately-sized" effect size. A lot of other things can account for the extra 30%, including picking the more dubious of findings out there (which seem to be mainly social psychology not cognitive psychology studies). Looking at your other comments on HN, you seem to love throwing bombs and trolling.

That isn't actually even close to what you said (very contrasting statements). 50% is quite horrible (I was not the one who down voted you, but I see others agree). BTW, mentioning your opinion of someone's "other comments" doesn't help your case either. It makes you appear desperate to cause a distraction or detraction.

Why don't the peer reviewers demand higher standards?

Inability.

One of the foundational problems with the soft/social sciences is that many of its practitioners have such weak analytical thinking skills (Note below) and the educational requirements for Psychology majors typically just include a few semesters of statistics courses designed to be understandable for them.

Peer reviewers in Psychology generally don't expect authors to go much beyond the basic statistical tricks students are taught in undergrad. From prior discussions with professors in these fields, it's my impression that most are unable to understand just how flimsy such methods are.

Personally, my prediction is that this won't get better until we have more AI-guided analytical analysis software packages that make more robust analysis accessible to those in soft-science fields. This is, something to replace reliance on stuff like R^2 and p- values.

---

Note: Just to provide some numerical evidence for the assertion that Psychology practitioners tend to be weak with their analytical skills, here's [a PDF of GRE scores](https://www.ets.org/s/gre/pdf/gre_guide_table4.pdf). GRE test-takers planning to study "Psychology" had quantitative reasoning (QR) mean of 149. This is pretty bottom-of-the-barrel; even people planning to go for grad school to do "Arts - Performance & Studio" did a little better, with an average of 151.

"One of the foundational problems with the soft/social sciences is that many of its practitioners have such weak analytical thinking skills (Note below)"

Andrew Gelman argued that a fundamental problem with peer review is that it is done by peers. Peers whose background and skills are similar to those of the authors, and thus not likely to catch things that were missed.

[When does peer review make no damn sense?](https://andrewgelman.com/2016/02/01/peer-review-make-no-damn...)

I'm sure something could be gained via education. I don't think you necessarily have to be good at math to understand the concepts. But a lot of undergrad courses focus on hypothesis testing and p-values (despite those methods being condemned by the American Statistical Association), and encourage memorization of steps to do simple math, over understanding what any of it means. I think the ad-hoc approach of many machine-learning intros would do better. Maybe programming isn't any easier for a psychology student, but simply hammering the ideas, with short demos of pitfalls, may help.

For "small n" problems like in psychology, psych students are much better off with a statistics background than machine learning. So what I'm advocating is a change in how these classes are taught.

Of course, I've known plenty of students who program via copy and pasting code, and modifying it until it runs as necessary. The equivalent of memorizing steps math steps. So it will take more to solve the problem.

The GRE is nonsense anyway so I wouldn't really take it as meaning much of anything. Just look at how awful the subject tests are to see why nobody should ever give GRE scores more than a passing glance.

I'll admit that might have something to do with it, although psychology GREs are misleading because a lot, probably the vast most majority, of those test-takers are hoping to go into clinical psych, especially to practice therapy. This leads to issues in a number of ways: (1) many of those applicants really don't understand what psychology is about, (2) most (90% in research-oriented programs) are rejected, and (3) among what's left, it has to be said that they probably don't need to be the most quantitative to be a good therapist, if they're not doing research.

Psychology is a strange science in that it's a mixture of people with very unquantitative backgrounds, and those who deal with very complex math and statistics. What many don't realize is that meta-analysis itself really was developed as a method in psychology (even if it technically has its origins earlier with Pearson). This registered replication work is an extension of that, again being done by psychologists. It's probably safe to say that more empirical and statistical research on the scientific process itself has been done by psychologists (along with statisticians and many public health researchers) than any other discipline.

In any event, replication problems happen in other domains as well. This has been documented empirically. It might be worse in the biomedical domain than, say, physics or chemistry, but it's not limited to psychologists. What I see in the neurosciences per se is just as bad, if not worse (because it's ignored more).

I think ignorance of issues pertaining to overfitting, etc. definitely contributes, but I also think that ignorance is pretty widespread, and the problems can be sort of pernicious in that they don't always operate intuitively.

> Why don't the peer reviewers demand higher standards?

I'm not sure peer review can change a field's direction. Say the top journal in a field used to publish the top 10% of papers. If only 1% of papers meet some higher standard, then if the journal chose to enforce this, they would essentially cease to exist.

And the people who would otherwise have got hired/promoted/tenured based on publications there, will now win these tournaments based on their publications in the second-best journal. Their contests are with others in the same field.

I guess the more hopeful scenario is that, besides the 1 of 10 top papers that are actually solid, there are (somewhere) 9 other solid papers with less-flashy results -- presumably carefully proving things that everyone thinks, not counter-intuitive things which get you a TED talk. Here it's more complicated, as if the journal chose to publish these instead, it seems to me that hiring/promotion/tenure committees may simply start to view it as a less-prestigious venue.

I think the fact we're even discussing a "replication" crisis points to some small movement in academic research to address this (some journals are changing / have changed policies as a result).

There's probably different causes at different times. Some of these effects that are the target of replication tests are relatively older, when people were less aware of some of this phenomena. To be fair, some of this stuff is unintuitive: for example, some of the major journals would require internal replications, over several samples, the rationale being that if someone shows an effect in 5 samples with slightly different designs, it's probably "real." It's not like some of these things were just based on single samples (although some of them certainly were). Of course, now people are aware that 5 small samples does not large-sample replication make.

My intuition is that another cause is that academics is flooded with researchers collecting a lot of data, under a lot of pressure to produce positive findings to attract money. Hype, TEDtalks, grants, hypercompetition, and so forth. Academics now is horribly incentivized to be popular and bring in money (from peers, it's important to note), rather than to be correct. Add in a complex subject, like human behavior, and it's a recipe for disaster. Academics is also full of conventions, that attain the strength of power structures; when these are baked in it's hard to change them because you have people who attained power under those conventions in charge (nothing conspiratorial really, just people have their biases and blindspots).

Yes. As a practitioner it’s very frustrating

I think the situation is pretty analogous to other area with replication crises like psychology and nutrition: local variables dominate. Just like our own body’s there’s a lot that just depends on the domain or the specifics of the problem being solved...

This has been my experience in a related field (information retrieval). There are trends and best practices, but the market and Impossible to recplicate overhyped academic research reinforce each other.

It's for this reason that I like publishing straight to my blog: http://statwonk.com/weibull.html Let the code / math do the work.

I intentionally make it easy to pick up, modify and inspect. The benefit to me is precisely being alerted if it isn't replicable.

So a replication crisis in artificial neuropsychology.

You may be joking, but I'm tired that everytime neural networks comes up this unfortunate conflation of different aspects of AI occurs; one based in accurate biological modeling, the other based in tractable statistical methods.

Whereas it is true that both are inspired by neural activation and that both can produce a complex, yet coherent mapping from input to output, this is like saying that birds and aircraft are similar because both fly and have wings.

Yet both birds and aircraft can stall. (Granted, birds can get out of stalls much more easily than aircraft can.) While many analogies between natural neural systems and various ML arrangements are false, not all of them are.

The analogy is like "both a human and a baseball launcher can throw a ball, so they must be the same." People really need to let go of the terminology around machine learning that leads people to really believing that it's close to actual cognition.

But what is cognition? People should reevaluate their views on how it works.

The brain is very complex and a very big ball of highly specialized circuitry (and the software running on those and managing various aspects and parts of it and them). The similarities are at the same time very striking but the differences are just as drastic in number of layers and other size related parameters our various brain components have.

Yes, AlphaGoZero is not going to learn to cook and sing and dance, but it beats our gaming component in a lot of areas. Of course a synapse is not really a memristor or a ReLU, but at the same time not that far off either. Similarly a biological neuron is not just a simple backpropagatin integrator like a perceptron, but it works very similarly.

And just as we don't really know how all the representations work in an ANN, we don't exactly know the biological aspects of memory/learning/seeing work in our brains, yet we have made enormous progress on both. See all the demos/visualizations on how various layers encode in deep nets and look at the data about the brain's visual cortex and the V1 circuit, look at the gene spliced mouse studies where memory encoding is studied (and sometimes only one neuron encodes for a memory/face/concept - just as with deep nets).

Indeed, they have similar characteristics in certain respects, and sometimes ML is inspired by neurobiology. I suppose my exasperation is primarily directed at non-engineers/researchers hyping AI; to abuse the analogy, sometimes these futurist writers seem to think that just because both can fly, aircraft can one day lay eggs too.

Why isn't more research and research materials open source in the AI world? I don't really understand.

If one doesn't have your training dataset or your code, how could they possibly replicate your results?

So the argument is that running buggy code makes your replication tainted. So ideally experiments replicate everything, meaning part of replication is implementing the code & collecting a dataset. The exact code & dataset aren't suppose to be required for the results: you should be able to replicate the results by substituting your own code & data. When you have a rat maze experiment, it isn't expected that they include a 3d printer blueprint of the maze & genomes of the rats involved

Or, more succinctly, code & data are left as an exercise for the reader

It is common to publish the layouts and dimensions of mazes in such studies. The mazes[1] themselves often reflect the nature of the study (e.g. a simple fork to test desicion-making or a complex maze with dead ends to test navigation and/or memory.)

1. http://www.ratbehavior.org/RatsAndMazes.htm

But if more journals archive artifices (code + data) and there was a bigger push to keep these around, when replication fails, they can at least go back to the exact code and say, "huh .. they implemented this differently. I think this is the problem with their algo" or "oh, we forgot to account for this." .. ideally after you've written your own, looking only at the methodology and without looking at the code for the original.

Having the journals archive artifices seems like a good idea, but the librarians hate it. The last thing they want to do is to give the for-profit journals another thing to monetize against the academy. What makes more sense is to have the universities and the library systems archive that stuff.

why not use a public artifact repository? e.g. github, bitbucket, or a new academic collaboration.

I see. That does make sense, like making a MIT version of GPL reference code.

Having more researchers avaliable to audit code seems like it would help prevent flaws from slipping through, too, to prevent false conclusions.

Thank you for explaining a bit more.

Research is often funded by firms that want the study to serve as a “scientific” basis for the efficacy of a product or service. Experiments that cannot be replicated are meaningless. That standard should not be compromised.

Amusingly, this is often taking to the other extreme. Just being able to run the exact same code on the exact same data doesn't really tell me much about if it replicates anywhere else.

That is, I agree with you that the code and the data should, ideally, be available. I lose confidence when people just rerun the same code on the same data. The slides a while back about why someone didn't like notebooks resonates well with me. Something like "Shift enter through the lesson. Is this learning?"

Are the 30% of papers including training data actually hosting the data set somewhere for download, or are they referencing a public data set elsewhere? Both approaches are acceptable in my book.

If a study uses a private data set, or one that the researcher controls and only gives out to approved partners, that study should be discounted.

I understand corporate labs cannot give away their data in many cases, but corporate research carries less authority than academic research anyways. Academic research should always make the data set publicly available.

I think if you're going to do a replication study, you should collect a different data set about the same subject, and use the same method, then see if you get the same results.

Yes, simply compiling the provided code and running it against provided data set does not really count as replication.

Writing your own code and creating your own dataset is the simplest way to rule out the situations where there is something fishy in either the original code or data. And the paper itself should contain enough details to make it possible to recreate the experiment this way.

I think compiling the provided code and running it against the provided data set does do something---you know the code reports on the data. If you get a different result with the provided code and data, then there's something different with your environment vs. the original researcher, perhaps the rounding mode, or some assumption [1]. Once that's straightened out, then you can use new data and see if you can replicate the result with the provided code and new data.

Think of the provided data as a sanity test.

[1] I recently fixed a bug wherein I was inadvertently relying upon Linux-specific behavior that failed when tested under Solaris.

If you have a good paper with important result, providing code and data is not necessary.

Providing the code to replicate is good form. It shows good faith and confidence. Exact replication (exactly replicating the study) is just the starting point to check that the code works and no obvious mistakes were made.

replication / reproducibility / hyperparameter sensitivity

If the research yields something really important and the method is well documented, usually it can be easily checked without having the data and the code. Things like dropout, batch normalization, residual learning, .. work over multiple different datasets and hyperparameters. You can reproduce the results without faithfully replicating the experiment.

If the claimed result vanishes unless you have the exact data, code or the hyperparameters, the research can't be said to be meaningfully reproducible in the scientific sense. Hyperparameter sensitivity is ML equivalent to P-Hacking.

how many papers important results are simply bugs?

Numerical code is already bug prone due to subtle and hard to test errors, research code that's not code reviewed or necessarily even tested for correctness can easily generate important results erroneously.

It's also counter-productive not to publish the underlying source code for these papers, as it adds a barrier to other researchers applying the algorithm in new situations. I'd be interested in seeing if those 6% of papers which include the code get more citations than the population of papers which do not include code.

> important results are simply bugs

Probably none. If the paper is important and collects citations, the algorithm is in use. Computer science != working code.

Code is required when you produce something where the scientific importance is less clear. There is need to provide more evidence. Many papers are just "Hey I made some some tweaks and it works in this particular case." Those papers should have working code.

Most papers leverage a results table which compares the newly proposed approach with existing approaches. This section is baselines the results of the new approach with prior work and helps determine whether a new result is actually important or yet another way to achieve the same results as previous work. e.g. the tables on page 7 of this paper https://www.semanticscholar.org/paper/Automatic-Acquisition-...

These tables are generated using real implementations that may or may not be correct, and should be subject to review when the paper is published.

As someone who's often implementing models from ML papers, the issue is that english-language descriptions of methods is often found to be sorely lacking. Even good authors simply forget to mention small critical details of model design or training that wouldn't remain ambiguous if they simply released their code. Part of the effort of reproducing work is certainly figuring out which aspects of the model design are critical or incidental - but this is all greatly aided by not having to guess at what was actually done from a heap of english and LaTeX generally written feverishly over a few days before a deadline!

> Part of the effort of reproducing work is certainly figuring out which aspects of the model design are critical or incidental

Isn't this the work of writing up? A paper is a claim that you have discovered something, and implicitly that other details are standard / unimportant. If it turns out that some hidden assumption was in fact doing all the work, then the claim you made was false.

I'm all in favor of sharing working code. But working code which magically does something... amounts to an anomaly awaiting an explanation. Or an advertisement.

Alas, training times are quite long... At any given time, there's probably a couple best-in-class architectures out there for the problem you're interested in, and one or two dozen interesting bells and whistles one can add as decoration, each with a paper that makes pretty reasonable arguments and has some stats demonstrating modest gains.

The right thing to do in this circumstance is an ablation study - throw together your best-possible model and then test different subsets of features sitting between your model and the 'basic' prior work. For large datasets, though, each of these models might take a very long time to train (especially if you don't work at a place with a stupid number of GPUs available).

So, lacking resources, you get your new best-ever accuracy number with your 'everything' model, and do an extensive write-up about how awesome the new bell and/or whistle that you added to the pile is... (The problem is compounded by a need to publish quick, lest someone else describe your bell/whistle first.)

Another big problem is that adding a bell/whistle to the base model often means adding more parameters to the model. There's decent evidence coming out of the AutoML world that number of parameters matters a hell of a lot more than how you arrange them. (It's real real easy to convince yourself that your clever new idea is more important than the shitpile of new parameters you've added to the model, after all.) So a really solid ablative study probably needs to scale the number of parameters in a reasonable way as you add/remove features... And there may not be obvious ways to do that smoothly.

And this is closely related to the replication study in psych: it's real real easy to do kinda sloppy work with big words attached, and convince yourself and all your peers that you're a genius.

I think a big database for reporting and searching for results with various architecture+dataset combinations would be much more useful than pushing more papers to the arxiv in many cases. (though, really, whynotboth.gif) Let me do some searches to see if a particular bell/whistle actually adds value across the god-knows-how-many-times someone's used it to train up imagenet from scratch...

Then it sounds like people are doing something quite far from science. That's not a criticism, there is lots of knowledge which isn't scientific... for example if you want to know how to run a restaurant, you need to talk to lots of people who do, and try to learn what they know about the art. Maybe you even go to trade shows and listen to talks. But if you follow what they said and you fail, it's not really a replication crisis.

It's a mix; some groups are more disciplined than others (and have better resources for ablation studies).

To be sure, some sciences are more rigorous than others. Various natural sciences are reliant on transient observations, which might or might not be 'reproducible' in any sense... And still progress is made, though the results might contain more prejudices than one would encounter in other areas.

It's also worth noting that some areas within machine learning are extremely difficult to test rigorously (e.g., generative processes), but are still totally worth pursuing. So be careful with cries of 'it's not even science, man!'

Right I genuinely don't mean that as an insult, perhaps my cooking example was flip. There is lots of genuine progress made by other means -- anyone think that Toyota's process inventions in were done by double-blind trials on a large sample of factories?

And of course some guys a few doors down are proving theorems. And lots of other things.

It is slightly strange though that we try to vet all of these with the same peer review mechanism. This is one major source of the differing opinions in this thread about how this ought to work, I think.

Reproducible builds make it much easier for others to check for things like hyperparameter sensitivity. Right?

Yes.

There is some famous citation from a physicist about people that used to replicate each other experiments before, but now they share their fortran models, so they can agree on all the bugs.

Yea, I have always found academic machine learning papers to have very little value. The helpful people create a github and write a blog post, not publish a paper.

Self-publishing code repositories or notebooks is clearly the superior approach, but unfortunately researchers are given less institutional credit if they are not published in an academic journal. The best approach is to get the benefits of peer review and then publish all your research artifacts alongside the paper. I am not sure if the academic journals prohibit this kind of thing though.

I'm pretty certain PLOS allows and even encourages this: https://www.plos.org/open-access

Elsevier is by far the worst publisher regarding open access though: https://twitter.com/protohedgehog/status/1028819653982736389...

Because you are not the target audience for these papers. ML researchers (or any researcher for that matter) don't have time to go through a long form blogpost or skim through a GitHub repo. There are tens of interesting results that come out everyday. Papers with abstract and sections is the most efficient way to understand the overall idea.

Blog post are nice for reaching outside of the research community.

Code should always be linked for people that want to reproduce, but they don't replace a paper.

>"ML researchers (or any researcher for that matter) don't have time to go through a long form blogpost or skim through a GitHub repo."

I don't see why this would take any more time than properly studying a paper, in fact it should be quicker since the information is presented in the proper format (not translated from code to math/prose).

Because you don't "properly" study all papers that you read.

For most, you just want to know the general ideas and understand the novel techniques that were applied. You might decide to do a deep dive in a very select few. At this point code becomes useful for reproducibility.

Also once you are experienced in the field, you don't relearn everything for every new paper. Most of the paper just propose incremental minor changes. The only things I care about is what these changes are (conceptually, not a code comparison) and data about the effect of these changes. I don't want to have to read your code to find out where you have applied a certain new hyper-parameter to your gradient descent.

Stop thinking that the entire research population is publishing all results as papers because they don't know better. It's indeed the most efficient way to share knowledge for experts in a given field.

> since the information is presented in the proper format (not translated from code to math/prose)

Do you really think researchers just write some random code until they come up with good results and then try to retrofit that with some equations?

You have got it completely backward.

>"Because you don't "properly" study all papers that you read.

I have much experience reading research literature (but not ML specifically). The vast majority of the time I just skim a paper looking for certain info.

What I am reading from you is that basically the only useful part of the ML paper is the abstract. If the repo/blog had a good abstract there would be no point to the paper.

>"Do you really think researchers just write some random code until they come up with good results and then try to retrofit that with some equations?"

This sounds like a strawman, but yea I do think the code is the "actual" method while the math is some idealized cleaned-up translation.

Engineering researcher, but not-ML.

"Code is everything" approach presumes that communication is computational by default. I'm not sure if researchers agree on that. This is particularly important for a field that aspire for artificial intelligence. Language is the best bet we have at the moment.

Secondly, there are social aspects. I am becoming more well read in my field, and there are time when genuine "rediscovery" occurs. Many phd students, depending on their research group, do not come up with ground breaking work right off the bat. It takes them a few years. In the publish and perish economy, there are venues to show your paper. If their genuine work is rejected, it may stop them from progressing in their career to come up with great work. It is like expecting an undergrad to come up with a full master's thesis for a course project. Happens, but not often.

Above being said, now if I am to read the code, I will be rereading many code repretitions every year. Whereas reading similar abstracts is less time consuming.

Your blogpost comment touches a bigger issue of how to tell who conducts legitimate research. The best we have so far is that those at the top of their field to provide assessment. They're journal editors. They have dedicated their life to their field, and have read paprs from decades of research work (whether or not the field is scientifically paramount is irrelevant, not all can learn the same thing, and education system is there to _educate_ the population at large, along with generating new knowledge, and along with advancing new researchers). For the sake of completeness I'd add this, at least in my case and perhaps many others, as one becomes more experienced, one can assess their earlier grasp of the field, or their earlier misunderstandings, better.

Hence it seems reasonable to publish, and have abstracts.

> What I am reading from you is that basically the only useful part of the ML paper is the abstract. If the repo/blog had a good abstract there would be no point to the paper.

The whole point of a blog post is vulgarization (which is definitely needed as well): explaining an idea without assuming that the reader has a lot of prior knowledge in the domain. Which goes in the opposite direction of what researchers look for: the most efficient way to understand what is being proposed. Most of ML papers abstract would be pretty horrible on a blog post in term of vulgarization.

Plus, papers are all very well centralized in Arxiv. How am I supposed to find blog posts? I should just hope that what I am interested in will blowup on twitter or /r/machinelearning and I will notice it? This sounds extremely sub-optimal.

> This sounds like a strawman, but yea I do think the code is the "actual" method while the math is some idealized cleaned-up translation.

Do you also think that biologist should stop writing papers and just upload video of their actual experiments?

I work in an ML research lab with 30 or so researchers, research never begins with some code snippets. It always begins on the whiteboard. Even between us, where we have the code and data readily available to share with each others, when someone presents it's work, they will never show code, but always equations or model diagrams. It's just an order of magnitude easier to visualize.

Take this paper for example: https://arxiv.org/pdf/1805.11604v3.pdf how would you even begin to communicate something like that through code?

(A small note on your interchangeable use of reproducibility and replicability in research because they mean slightly different things.

reproducible: independently achieving the same result using existing data;

replicable: independently achieving the same result using new data.)

At a recent Boulder deep learning talk a distibuished scientist asked the speaker 'how can I use your result?' There are not easy ways to publish and distribute such results / models.

Is there a video of this talk?

I’ve raised the issue of pseudo code and missing data multiple times in my masters work. My contention is that public funding should require public access to such information for external verifying. I’ve been shutdown on every front. People say that the private work needs to remain private in order to ensure future research monitization from grants and IP sales.

Reply on news.ycombinator.com