I imagine those loud objectors would take issue with the idea that the government is "literally just US" - it's much more literally a small minority of us who are deciding on behalf of us what we are to do with our resources. There's nothing inherently wrong with that - the US was always intended to be exactly that - but it certainly leaves room for the represented to debate about the way they should be represented. "Us" consists of people with different ideas about what is worth spending money on, both in terms of goals and effective ways of pursuing those goals, and using the government to allocate funding ignores all that.

It's easy to look at Tesla and SpaceX and see a couple winning tickets in the "give money for technology" lottery, but they are not the only tickets our government has purchased. Consider all the other entrepreneurs who are doing an outstanding job of not so much moving technology in any direction at all but rather lining their pockets while cratering their industry's reputation, or those that make reasonable attempts with interesting but ultimately impractical innovations.

The people who object are generally objecting to the playing of the lottery itself. The most commonly proposed alternative is that these companies should earn the money by providing value in the market or convince investors (who have powerful financial incentives to get it right) that they can do so, rather than by convincing a relatively few laypeople with no real skin in the game to risk other people's money on them.

Pre-emptive disclaimer: yes, I know I've not mentioned any of the opposing arguments here. My purpose in writing this is solely to explain some of the key points (including some of the emotional appeals) in this particular side of this particular debate.