This is obviously an overly simple narrative, but one real factor: the Bell Labs model was built around giving brilliant people a lab and a bunch of funding, and leaving them on their own to explore for a while. Lots of blue sky goofy research, a lot of which ended up being useful. That has its own problems, including “which few brilliant minds get this opportunity?” and “how do we make the researchers accountable for actually getting something done?”

These are both reasonable questions about equitability and accountability. Unfortunately the solution we chose is a proliferation of bureaucracies that micromanage funding allocation and use. Some widely acknowledged consequences are 1) researchers spend more and more time writing grants and reports, and less and less doing research, and 2) the funding agencies (public and private, but especially public) are conservative and overwhelmingly fund work that they know will succeed. In practice that encourages monothink and endless incremental improvements on things that we already know how to do, and disincentives dissent, creativity, and real blue sky novel ideas.

Everyone loves to say they support creative ground breaking ideas, but that requires letting smart people sit around and think for a long time. And however smart they are, results are not guaranteed. The bureaucratic process is always going to prefer short term thinking with clear “deliverables”, even when it’s detrimental to progress.