It's a direct side effect of reputation and funding being closely correlated: if your reputation is that you put out stuff that doesn't work you won't get funded. This is dumb, but that's how the world works. That's why you almost always see the 'more research is needed' line in various papers, it is most helpful when seeking for funding that one paper will lead to another. But (unfortunately) negative results aren't nearly as often published, and that is because they will not get cited as much in follow on papers. It's all the result of metrics based meta analysis of papers, aka the 'impact factor' (which, no kidding is a copyrighted term), once that got established that became the thing that science partially optimized for.

During the 'golden age' of science, the time of the Royal Society the fields weren't specialized at all and the publication mechanism was scientists sending each other interesting stuff by post. At that time there was no meta analysis at all and there was so much low hanging fruit that the 'gentleman scientist' could make big breakthroughs in their home laboratories. But as that low hanging fruit decreased the educational paths required before being able to do meaningful science became longer and longer, then specialization set in and the costs of doing science went up. That's how we arrived at grants used to fund science.