I was thinking about this last week[1] and I think that both the "math" people and "common sense" people are correct in a sense, and talking past each other. The math people are of course mathematically correct within the limits of the constructed model, given all the assumptions of perfectly random sampling, no systemic error, etc. Meanwhile common sense people are correct in a practical sense: small samples are vulnerable to sampling error, p-hacking, outright fraud, etc. Even before you're aware of the exact mechanisms by which things can go wrong you intuitively know that small/cheap studies are more vulnerable to some kind of honest human error or dishonest... shenanigans.

---

[1] I was listening to a podcast where trolley problems were brought up and the speaker was lamenting how clearly "unethical" and "irrational" your evolved intuition is given that most people will let the train hit 10 men working on the tracks than to divert it and kill 1 innocent. Trolley problems are intellectually interesting for various reasons but jumping to that conclusion is clearly absurd. Your intuitions are shaped by millions of years of genetic and social evolution to precisely be most rational for actual real-life problems. If you were actually standing at that switch you'd be thinking...

* do I actually trust my eyes in this situation? Are the workers on a parallel track and there's no actual problem here?

* if I pull the switch, will it derail the train and kill N+1 people instead of the 10.

* will the workers just notice the train in time and scurry off the track? Or will the train just stop? How good are brakes on a train anyway?

* how much time do judges and juries spend solving trolley problems?

... and while you were paralyzed thinking about these and a million other things, whatever was about to happen would happen and there would be no trolley problem.