Why isn't reproducibility required to publish in CS? Unlike in fields like psychology or chemistry, reproducing results should be trivial if the authors provide instructions on how to do it.
Why isn't reproducibility required to publish in CS? Unlike in fields like psychology or chemistry, reproducing results should be trivial if the authors provide instructions on how to do it.
Applied to CS, there's really no way around it: reproduction requires that researchers claiming to replicate a result implement it independently. If there are not two independently produced implementations that both confirm the result, the research hasn't been reproduced. The process of doing so helps to discover cases where the original results were due to idiosyncracies of the original implementation, test setup, etc. It'd even be ideal if replication was done in as dissimilar a setup as possible, to find cases where the results unexpectedly depend on details of the original setup not thought to be important.
If anything I think there are quite some dangers that reproduction will be decreased by the current trend towards what's questionably called "reproducible research" as a euphemism for "code reuse". If people reuse code rather than doing their own independent reimplementation of methods as stated in papers, erroneous results can lurk for years and infect other research as well. (I think code reuse is good for engineering practicality, and making permissively licensed code available is also a good way of getting researchy methods out of academia into the real world. But I think it is quite wrong to call reusing the original researcher's setup, rather than independently producing your own, "reproduction" in the scientific sense.)
True reproduction remains necessary to confirm that the stated underlying science is the cause, and not some other variable that might be legitimate and remarkable but undocumented.
The thing that is interesting or novel about this protocol as opposed to the thousands of other proposals is that it's supposed to be faster than the other proposals for a broad range of applications with a broad range of link topologies and across a broad range of software/hardware platforms.
Of course, to establish that claim replication is completely critical.
In what sense is a functional implementation not able to do real work?
I once went so far as to email the author of a paper to ask for their implementation of a Minkowski 3D hull algorithm, and they point blank refused. Now I don't believe I'm entitled to this of course, it's just strange you can claim something in a paper and for it to not be reproducible.
Oddly I think it's the triviality of sharing implementations that make them more guarded.