Even for real reproductions, that require work just like they do in any other scientific domain, the points in that issue seem pretty irrelevant. There's things like FAIR standards on reproducibility but as far as the status quo goes that repo doesn't look too bad. I could not care less if tests for some project are badly written, at least it's in written in a non-obscure language and shows a somewhat sane structure. What's next? Calling for redactions because somebody didn't follow the same tabs vs spaces paradigm?
There's nothing in that issue w.r.t. whatever scientific finding this was used for and this general phenomenon is really fascinating. Instead of a constructive discussion with the technical or scientific folk that put their work out there, or engagement with the politicians that drew conclusions based on those scientific findings, you get github issues and Twitter threads mixing a bunch of unrelated concerns.
Maybe a very dedicated reviewer might check a short simulation code that can be written in a few hours max, but even that is exceptionally rare in my experience. Reviewers usually look at the results and validation simulations in the manuscript, and if they look reasonable trust the code doesn’t have major bugs affecting accuracy.
Even if it is re-implemented and verified it won't be the same. There are a million possible conditions for "re-implementation". Where would you draw the line? How many "re-implementations" and "types of conditions" do you need to be sure?
You probably need to have expectation mismanagement from a scientific paper. A scientific article will only say "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions". That is the crux of ANY empirical science - Computer Science or otherwise. That's all that they are paid (and incentivised) for. A computational scientific experiment is not a software product. If you want more, you need to pour in more funding and give them more resources explicitly for those purposes. Either that, or if you want to be 100% verified - you can choose to read theoretical papers where mathematical proofs are "verification".
> "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions".
Is what I'm calling the assumption (the whole sentence). The implementation is essentially the methodology, and IMO therefore needs to be included.
Note I'm arguing against 'academic, paper-supporting, code does or should not need to be open sourced', I'm not saying it needs to be 'software product' quality, written in the same way, to the same expectations, packaged, or anything like that. Just available for someone to say 'wait a second, you didn't try it under conditions, X,Y,Z, because Y gets negated here', or whatever.