— Isaac Newton
In many respects, math has a totally unfair advantage over every other subject you could possibly study in college because it can be learned almost without needing any other people. Well, living people that is, as you do need the books of long deceased thinkers. (I'd say this also applies to philosophy, and hardly anything else). You just need to be willing to spend enormous amounts of time alone in contemplation reading.
The sole cause of man's unhappiness is that he does not know how to stay quietly in his room.
— Blaise Pascal
There's also a very limited social component layer such as you would find in medicine, law, politics, economics, et al. To become a doctor, or any kind of "professional", sure, it requires a massive social apparatus on top of a high degree of actual learning. You have to have connections, know the right people, say the right and popular things to them, be supported and credentialed by the leadership and their agenda, and basically imposture one's self into mimicking whatever behaviors and traits are specific to and dominant in whatever professional field you're studying. Eventually your identity morphs into this set of learned behaviors, and it's almost entirely built upon your position in some type of social hierarchy.
Not so with math. You either know some theorem and the proof, or you don't. How much you know and how advanced you are is entirely dependent upon what you have worked through on your own. What new stuff you can create only comes from what old stuff you already know. None of this has anything to do with interacting with people. Sure, you can benefit by talking to other mathematicians, attending lectures, conferences, teaching others, etc, but at the end of the day, at some point, you still have to sit down and learn the material by yourself. And if you don't do that, it doesn't matter how much of a social butterfly you are—you can't be a mathematician.
No man is an island, but I imagine a Robinson Crusoe type figure would have no such problems becoming a mathematician on a desert island, given enough math books(and some kind of Internet connection).