I've worked in research most my career and I tend to agree. The issue is that throwing more people in the same narrow approach most likely won't help if that approach is off to begin with. It has been promising and we've been getting and still get some returns out of it, but is it possible we're stuck in a local minima of the solution space in terms of representing the real system were trying to understand and manipulate? We of course can't know and those who are captains of a discipline and the funding agencies that decide who gets funded and don't steer to focus on narrow incremental changes. Some of this has to do with deeper business management steeping into basic scientific research, where I believe, it doesn't belong because it thinks in the wrong time horizons and has the wrong goals.
High risk, novel approaches often aren't funded or few opportunities exist for them. There is good reason for this because it can be abused by those just looking for fun or easy work while labeling it novel. It may also encourage fringe sciences that are bordering on psuedo-scientific work. They incorporate a bit of science but then go off in directions that are almost provably wrong. Distinguishing which novel approaches seem like genuinely novel realistic and non abusive proposals isn't always easy. In some cases they're just laughably wrong because there are so many assumptions baked in that are almost provably wrong or at least self inconsistent. On the other hand, some aren't quite so easy and involve a significant amount of effort and insight to understand exactly what's being proposed. This is often where paradigm shifting research really occurs.
It can take incredibly brilliant scientists with enough creativity to see the opportunity in a proposal and approve it and those aren't the people often awarding proposals. On the flip side, most of these sort of proposals, no matter how valid and novel they may be, simply aren't going to be correct and the reviewer is right to take a more critical eye. The novel approaches are inherently high risk and most will be wrong. I still think we need to fund these approaches. I've been involved in proposals that seemed only slightly novel in direction from accepted paradigms and reviewer responses came back (some agencies return anonymized responses) in a way that showed they clearly didn't understand enough about the domain to even make the assessment when their critiques were clearly invalid, they just didn't agree with the proposal approach. Maybe they were right the overall approach was off and created a lame excuse, maybe they were wrong, but this tendency to be risk averse even in the few funding opportunities that were clearly budgeted to be high risk shows the culture we have in modern science.
The burden of responsibility often lies on those proposing these huge shifts and we may be reaching points in some domains of human knowledge where the burden of proof is simply too high for an individual to provide for a given novel idea. Imagine if Peter Higgs alone had to provide evidence to his idea. It wasn't until so many other iteratative approaches were exhausted that particle physics decided they had to start testing novel other options. How much time, effort, even careers were wasted chasing other ideas? Should science be depth first search, breadth first search, a mixture of the two to try and hedge out bets that if we are on the wrong direction we might find a better paradigm in parallel we can jump to when we're stuck, or perhaps a different heuristic?
Then the really really difficult question, for me: if we support a mixture model of say BFS, DFS (and perhaps a handful of others) for the search space of acquirable knowledge, who should decide resourcing allocations for the search mixture model? What is the best set of approaches and resourcing? Science already incorporates complex search mixture approaches for knowledge at various levels but it seems at the highest level (how we resource stuff) it doesn't, it's incredibly iterative, risk averse, and focused on short time horizons for returns on investment.