When would they have the time and money to set aside to get deep into a subject that most of them do not have the academic background to see the value of?
Yes, everyone could learn mathematics. Just like everyone could get rich and go into space. It's just that for a huge proportion of people the pre-requisites are not there.
Ah the false analogy[1], we meet again. You don't really believe this do you? I mean you could make the same argument about gardening. Literally anyone can learn at least basic math, most people just don't have the inclination.
I feel particularly deceived about this state of affairs because my parents paid a lot of money for my formal education (school, university), and I got significantly less value from it than from people simply willing to share their knowledge for free.
Imagine me or someone else that was never part of the math research establishment came up with the exact same proof Perelman came up with. Ipsis verbis.
You can rest assured that at least the slack that they cut him on sketchy details would not be extended to an outsider. (In a way the Yau debacle was a manifestation of this)
There are a few examples of this already in somewhat obscure fields.
That's assuming anyone would even give the proof the time of day.
It might be worth thinking in this "math for and from the elites" mindset when describing public aversion to the subject. Seeing it as beyond oneself or not relevant to one's life lends little towards anyone's desire to learn anything about it.
It's easy to say "why don't they just do it," with some diffuse definition of "they" in mind, but even the most basic things can be hard to get for the poor. A proper maths education is one of them, I think.