IMO, programming is "easier" to learn on your own for a few major reasons:
1) The sorts of things most people are interested in building just aren't unforgiving intellectually in the same way that math is.
2) You have a compiler to check if you're right, and your code will often still work even if it's "wrong" (not as efficient as it could be, has unwanted side effects, etc.). In some sense the compiler is a bit like a teacher this way.
3) With programming, you can upload and make it available for free, and whether it's legit or not is largely disconnected from your pedigree or how "correct" it is. This makes programming far more accessible. This makes sense considering that programming is primarily a practical tool. On the other hand, mathematics is primarily a field of scientific inquiry and is judged by different standards. If you learn a bit of math, try to write a paper, submit it to the arXiv... well, people will probably think you're a crank.
On the other hand, if you're just interested in math for the love of the game... you can certainly pick up a book and read it, maybe work some problems, but I think at this point it's quite easy to fool yourself into thinking you understand more than you actually do. I guess there's no real harm in being a charlatan, but probably the average person is interested in having some kind of real relationship with mathematics that they can be confident has a firm foundation. I'm very skeptical most people can truly pull this off by just reading books and not actually going to school.
---
As an aside, I think the fetishization of math in programming communities is very interesting...