Music can be exactly as solitary and cerebral as math. That aspect of music doesn't get the same press, is all. If you play three chords on a tall stage people know you're in a band; if you sit in your home working out a Bach fugue nobody needs to know.
I have a slightly silly book about birdwatching. It has a section which talks about birding competitions - "Big Days", they call them - and the final two sentences stick in my head, always:
Don't go away from this thinking that birding is an endless succession of fabbydoo games or fun, fun, fun without a moment's rest. Quite the opposite. Birding is mostly about looking at birds. It is no use to pretend otherwise.
A useful motto.
Birding is very popular, of course, even in the go-go all-action United States. Other popular activities include knitting, gardening, hiking, sudoku, crosswords, reading, fantasy football, Angry Birds, writing comments on message boards, and, ahem, hacking. All of them solitary, cerebral, and socially unglorified. (Though I guess fantasy football is arguable: You can probably spend an entire evening over beer talking with your friends about the performance of their fantasy football teams. For all I know it's even popular in high school. I knew several people in high school who were reliable authorities on every active major-league baseball player...)
People have a warped view of other people's lives in general, and particularly the lives of high school students. I suspect that high school kids enjoy quiet, solitary activities as much as anyone else: In other words, more than you notice. The quiet hobbies are just unobtrusive, so they don't get a lot of press and they don't show up in movies or TV except around the edges.
Speaking for myself, I think I stayed away from band in school because it was full of people and attention. I gravitated toward the quiet sources of pleasure and meaning. Of course, I was blessed with a math club mentor who knew what "recreational mathematics" was...