"this seems patently false to me, you hang out with the people who energize and care about you whether you are 16 or 25..."

The sample of such people is MUCH smaller in high school than it is in the real world (easily five orders of magnitude smaller).

"but it really doesn't take up that much time,"

Four years full-time is a lot, especially when you're young. The average job lasts less than three years.

"but I would argue that going to college without at least calculus isn't the best idea if you plan to have a lot of time to learn all the cool math you want/need."

Agreed, but you don't need high school for calculus, and high schools usually do a terrible job of teaching it (how many people know what a derivative is five years later?)