What does "far from optimal" mean? Let's pretend the allocators are not god-like decision makers (if they are, let's fire all the scientists because our god-like decision makers can do the research for them).
Let's just accept that if we have a large bunch of diverse potential researchers, and limited funding, the best way to get a breakthrough is by funding a random sample of these researchers.
Maybe this sample should be stratified, but it should not be stratified in such a way that it incentives chasing grants (effectively making the grant-allocators the real PIs - with all their flaws and biases systematically driving the process, but without any accountability or management effort on their behalf).
We can't get optimal without unlimited wisdom, and if we had unlimited wisdom we wouldn't need scientists.
Counterpoint: the current system doesn't do this. It optimises for competency to attract funding. This occasionally overlaps wroth competency to perform good research, or competency to perform research well, or competency to oversee, nurture and guide research in others. But that is far from given.
The first pool is for idea generation, failure is an option. The second pool is to direct invention for social benefit.
So why not just fund them?
Any process you could create or that currently exists just ends up being gamed by people who understand how to get funding.
There's also the fact that a PI minting dozens of PhD over their careers, especially in fields with limited industry opportunities, inherently produces a pyramidal scheme that is bound to push people away from science.
We seem to have no problem with infinite funding for Department of Defense and to bail out banks. The amount needed for science is paltry by comparison and typically has much better long term return on investment.
The discussion on how much money should be allocated to research is a different one, and it's always going to be finite. In the end, funds for research as conducted in universities have to be extracted from societal economic outputs, be it from taxes or otherwise, which is sadly not infinite.
As for why research doesn't get more funding, sadly there is a faction in the US government that is opposed to universities and to government spending in general, and views research funding as easy to cut without provoking public outcry among their core voting demographics. This is the same reason that the Department of Defense is now such a large part of research funding, that same voting demographic is far more vocally opposed to reducing military spending.
These?
"Social Security and Medicare are sometimes called "entitlements," because people meeting relevant eligibility requirements are legally entitled to benefits, although most pay taxes into these programs throughout their working lives."
https://en.wikipedia.org/wiki/Expenditures_in_the_United_Sta...
And increasing funding will not increase demand in lockstep. There are indeed more people who want to do science then there is money allocated to pay them, but the number of those people is not going to vastly increase if 'researcher' continues to be a job that requires years of post-secondary education, has long, shitty hours, and pays peanuts.