> As a reviewer on open code papers I would usually try to run the provided code
You're only one of two people I've ever heard make this claim. Which I'm sure you're aware, but many people probably aren't. Fwiw, I'm often called diligent because I read the code (looking at main method and anything critical or suspicious. Might run if suspicious). Even reading supplementary materials will earn you that title (which is inane). According to this informal survey, ~45% of neurips read the supplementary material <13% of the time and less than a third always read it[0] (I'm in that third, and presumably xmcqdpt2).
> Usually peer review is more about making sure the work is novel and interesting
This is why I find p̶e̶e̶r̶ ̶r̶e̶v̶i̶e̶w̶[1] journal/conference reviewing highly problematic and why this system is at the root of our current existential crisis: the reproduction crisis. Reproduction is the cornerstone of science. And many MANY good works are not novel in the slightest. See the work of Ross Wightman (timm) or Phil Wang (lucidrains). These people are doing critical work in the area of ML but they aren't really going to get "published" for these efforts. Many others do similar work, but just not at the same scale and so you'll likely not hear of them, but they are still critical to the ecosystem.
But with your next point: if it builds, merge it; I'm all for. The system should be about checking technical soundness and accuracy, NOT about novelty and how interesting it is. Of course we shouldn't allow plagiarism (claiming works/ideas that aren't your own), but we should allow: replications, revisiting (e.g. old methods, current frameworks (see ResNet strikes back)), surveys, technical studies, and all that. Novelty is a sham. Almost all work is incremental and thus we get highly subjective criteria for passing the bar.
Which is probably why high impact journals are not considered the most accurate. Because they don't encourage science so much as they encourage paper milling, rushing, and good writing.
[0] https://twitter.com/sarahookr/status/1660250223745314819
[1] We need to stop calling journal/conference reviewing "peer reviewing." Peer review is when your peers review. Full stop. This can come in many forms. Similarly publishing is when you publish a paper. Many important works come through open publishing.