There are 144.3 million taxpayers in the US, and 329.5 million people. Using the still-accurate numbers of 8% of the population being in college at any given time, and the average college tuition in the US of $8,000/year, we can do the same calculation:
(0.08 * 329,500,000 people * $8,000/yr) / (144,300,000 taxpayers) = $1461.40.
In other words, by subsidizing higher education with taxes, every person in the country could attend college for about 1/5 the cost, as opposed to "only people who can afford at least $8,000/year can attend college, and it costs them 5x as much as it otherwise would."
Those are real numbers. Does that make it feel less "hand-waving" to you?