This is a great point, and I think it shows why people don't study math in their spare time. It's the same reason why most people aren't body builders with 5% body fat: it takes a tremendous amount of hard work.
Not only that, but the work isn't glamorous or particularly creative. In my teens, I always though I was "bad" at math -- I never did the homework, didn't do the supplementary exercises, had an "intuitive" understanding that wouldn't translate when exam time rolled around, and I consistently got C's and F's.
In college, I decided to take things more seriously and I remember in Calc I-III doing literally every single problem in the books. If I missed any problem, I would do it over and over again until I got it right. "Miraculously," I started getting A's in math classes.