Furious AI researcher creates a list of non-reproducible machine learning papers

This isn't necessarily a recent development either. I tried to reproduce Ghahramani's factorial HMM* directly from his 1997 paper, and ran into the same problem as the other implementation I found (basically, exploding mean field values or ill-conditioned covariance matrices).

By luck, I managed to find his original Matlab code and discovered he used a few small hacks (mostly re-scaling and reconditioning various values) to get it all to work.

I have no problem with that, but I simply cannot understand why you would omit these from the paper (though I can think of one reason - because it ruins your elegant algebra). There is no way anyone would be able to reproduce his results if the code disappeared.

https://github.com/nmfisher/fhmm_julia/blob/master/structure...

Let's start with getting code, then working code, before we start complaining that your new proposed novel method doesn't generalize well

This made me think of this excellent read:

https://www.newyorker.com/magazine/2010/12/13/the-truth-wear...

It's also common for papers with code to not generalize well at all.

This is probably why the AI field will undergo a new winter.

It’s not science if it’s not reproducible.