Problems of Reproducibility in Computer Science
blog.michaelpankov.com
blog.michaelpankov.com
We've built systems that are very fast, but almost no one knows why they are fast and when those speed advantages go against us (which is quite rare) it's often unclear why. For the most part, all of this has been a huge win. It might make reasoning about performance from first principles ugly, if not impossible, but the economics favor it. That said, we're at the point where the only way to reason about performance in many real-world systems is empirically, because there's just so much going on in terms of memory hierarchy, network traffic, etc. that is hard to predict in advance. Combine this with the myriad variables, some in the physical world, that can influence performance... and what we end up with is that results that seem meaningful (30 percent faster!) might be artifacts of differences in configuration.
That said, I don't think the term computer science was coined with performance measurement in mind but, all that said, the field is not really a science. It's a mix of mathematics, microeconomics, design and engineering... with a lot of science involved, but most often not scientific under the original, strict definition of the word. Science is about investigating the natural world and finding the right answers; CS is about making answers. And they're both extremely important, but fundamentally different.
Also, just to underscore how difficult CS is, I tend to think of it as a branch of mathematics that lives on an unfair playing field. Reasoning about arbitrary code, in even the most trivial ways, is known to be impossible (Turing, Church, Rice) and much of CS is contending with the obscenity of arbitrary programs (i.e. often having to quantify over the space of programs) and, typically, that means limiting what can be written while still retaining as much power as is needed.
It's pretty simple, all code can be modelled as a fluid flowing through a landscape. i.e. imagine a river (which represents code), it courses over the land and there are "cliffs", rocks and hills (representing bottle necks/loads).
The reality is as bus's got wider and transistors were able to be packed closer, it just allow for more water to flow faster. There's no big conspiracy at all. Because software ultimately can be expressed as a circuit.
No one (to my knowledge) is suggesting "conspiracy" or malicious intent, or even incompetence. Hardware designers know what they're doing, and on the tradeoff between maximizing performance vs. making performance simple, the former is usually the best option.
So you either need
1) a smarter approach 2) focus on whether the problem is tractable or not 3) If you're going to be ambitious you're going to have to pay for it.
Human beings aren't really known for their wisdom however.
Secondly, testing in a number of different environments is part of the nature of scientific reproducibility, to check if the claimed result is robust or simply an artifact of the experimental setup, so releasing more detailed specs of the hardware used is not so essential.