That is not correct. Something like the bottom 50%(?) of earners, including all of the truly poor, do not pay taxes at all, but they benefit from the spending done on their behalf -- in some cases through directed programs, and in other cases just like all citizens.
There is a stratum of taxpayers who do not benefit directly (i.e. loan repayments) from government debt because they do not hold bonds. And as you say, in the upper economic strata, bond holding is more common.
So the strata that are least-benefitted are somewhere in the middle to upper middle class (such as it is). But these people are mostly OK -- their needs are met, most of their wants are met, but they could optimize better. NBD.
AND: If those strata did optimize better (move some income/savings into govt bonds), then the govt could borrow more and everyone would be even happier except the folks who obsess over the number for political reasons (when political party "not-mine" is in office).
Of course there is a "donut hole" in the middle where the social benefits available (programs) are inadequate, and the economic benefits (bond holding) are unreachable.
But the proposed programs which are running into this wall of discussion are in many cases designed to address this gap. There are good arguments about how best (effectively, cost-effectively, etc) to address the gap, how to design the programs.
But the argument that "we can't afford it" is short, pithy, and easily-comprehended, but much too simplistic to be honest. "We don't want to" is more correct.