"government is a slow moving, inefficient behemoth"
It most certainly is, but that's not relevant.
The important thing here is that so far only governments have had the capacity to make investments that are both of:
- High risk / high payoff
- Long term payoff / broad payoff
The second part is the catch. Private funding mechanisms can make investments that are high risk / high payoff as long as they are both short term enough to fit within private investment vehicle life cycles and narrow enough to allow value created to be monetized -- captured, in other words.
Things like basic research and fundamental invention are long term and broad -- they add value across wide swaths of society and across generational time scales. As such they really can't be monetized using any known financial construct. Private funding vehicles simply cannot invest in such things as there is no potential for private return.
Governments can make these kinds of investments because they are likely to be around long enough to see the return and because they can capture the return by taxing the entire economy -- in other words, they capture return by long-term economic growth and national strength/prestige.
Until or unless someone invents a private investment vehicle that can do this, we are stuck with government as the sole practical financier of the first steps in the technology chain. It is indeed inefficient, but an inefficient financier and generator of invention is superior to none.
Lately I've been leaning toward the view that "market fundamentalism" is partly responsible for the collapse of deep revolutionary invention after around 1970 or so. The timing is certainly right and the causal mechanism seems to be there. The symbolic point would be the cancellation of the Apollo program. By contrast anti-market fundamentalism is probably why the USSR invented a lot but never followed through on most of it. The ideal is to combine the systems. Fundamentalist dogmas are idiotic.
Edit:
Governments probably aren't that good at invention either. They just look great when compared against nothing.
That being said, governments are terrible at every step after fundamental invention. That's why Engelbart/SRI had to be followed by PARC and Apple and such, otherwise what they invented would still be a lab curiosity. They just don't have the mechanisms or institutional culture to "ship" or innovate.
Invention and innovation are not the same thing. Saying they are is arguing that all combinatorial fitness landscapes are smooth and "well behaved," and if that were the case machine learning would be a lot easier. Examples:
Invention: discovery and harnessing of radio waves, an absolutely unprecedented step change in technology. First we had no ability to communicate both wirelessly and invisibly over great distances, and then we did. Inventions are qualitative changes that open vast new regions of the combinatorial space -- they're analogous in evolutionary learning theory terms to journeys across physical space to land on a new planet.
Innovation: morse code radio -> voice -> high quality / FM voice -> television -> digital -> spread spectrum / TDMA digital -> ... Innovations are quantitative changes that are more akin to gradient descent in learning theory terms.
The evolutionary analogy extends both ways. There are inventions in biology too, like the appearance of photosynthesis or eukaryotic cell structures. These are unlikely events and therefore do not occur very often in the fossil record. Innovation occurs constantly.
Biology could invent more if there were a god who intentionally biased the system in ways that facilitated it, but as far as we know no such thing exists. But for cultural evolution -- the evolution of extended phenotype -- we are the gods and can indeed do that.
Edit #2:
I'm sure people will cite cases where independents or small academic teams operating on only small amounts of funding have invented world-changing things. Two things about that. One is that it was more common back when science was in its earlier stages -- as science has developed we have kept raising the bar in terms of how much capital is required to keep going. Today the capital and labor requirements to make fundamental inventions are (in most fields) far outside the reach of small funding vehicles or individuals. Second is that this is just plain uncommon. Even back then most of these folks had deep pocketed benefactors or were themselves wealthy. There are always exceptions but you shouldn't base policy on exceptions.