This is not just math, it's virtually every subject you learned in school. Imagine the following at a cocktail party:
"I had to read Joyce, Whitman, and Hawthorne, but I never got the point of any of them."
"I took a lot of history, it was a bunch of dates which I could never remember."
"I took three years of Spanish, but I don't remember a word."
Et cetera.
A lot of people just don't click with formalized education at all in their early years. They just couldn't throw themselves behind all those authors and facts they're "supposed to" be up on. Some of them are too busy learning informally by dealing with the world around them -- i.e. with things like nature, work, and you know, other people.
Then slowly grow to form their own view of the world, and seek their own path to obtaining a more structured comprehension of it -- which might well mean starting (on their own volition) formalized education sometime in their 20s or 30s -- or perhaps not.
A great many who choose this route, or have it forced upon them may fail or flounder -- but a great many also don't, and end up with accomplishments for others to read about or hear about in those books, classes and received intuition that some people think are the only valid starting points to figuring out how to make a dent in the universe.