In my masters degree I took a course where the entirety of the grade was in replicating a machine learning paper. It’s wasn’t a well-known one, but it still had many citations.

Well it turns out that the paper leaves out many key details in how it would actually be replicated, and the GitHub link was long since deleted (and not archived either). After many good faith efforts to reproduce the results we realized that the published result was actually probably due to the authors’ choice of a lucky random seed. Or more likely, rerunning the experiment many times with different seeds. In reinforcement learning that’s not uncommon.

Even if I am wrong, I still believe that a part of peer review should be in a third party being able to replicate your results, at least in computer science. It’s expensive but imo would lend a lot more credibility to results.