I think we should be more generous, given the fact that we cannot reproduce many findings in our own field (computer science) given the hardware to do so is limited to such few entities...
Sounds like we should be less generous to CS findings, not more generous to other fields. (And yes, we should be skeptical of findings in CS as well.)
> given the hardware to do so is limited to such few entities...
That said - what research is happening in CS that needs specific hardware? The theoretical stuff can still happen on chalk boards, and interesting algorithmic or technical advances tend to propagate quickly precisely because someone will reproduce them.
We really shouldn't.
> given the fact that we cannot reproduce many findings in our own field (computer science)
Computer science isn't a "science". Computer science is really a branch of mathematics. For example, when you study computation theory, you prove theorems (deduction). You don't generate a hypothesis and test it.
> given the hardware to do so is limited to such few entities...
Unless you are talking about computer engineering, which isn't really science either but engineering. Computer science isn't done in "hardware". Maybe you should go learn what computer science is.
For example, if you propose some new technique to make databases faster (e.g. "store tuples column-wise instead of row-wise"), you'll implement it and run various workloads with and without the technique enabled. That gives you a quantitative measurement of the merit of the technique.
The engineering artifact is just a by-product. More often than not, the code is thrown away or never used again, once the experiments have been run and the paper has been published.
I came to an opinion that most of the current AI research can be easily reproduced on the small scale. CoT is possibly the only exception as it sounds like it requires certain emergent behavior, but even there I am not sure it is impossible to retrofit to tiny models.