But what level of programming and mathematics are you comparing here though? because college-level algebra and calculus is really not that hard imho (once it "clicks" for you, but it's the same for programming), and if we are comparing math as in what you see in a BSc/Msc of Mathematics (or research-level) then I agree it's hard but you have to compare for an equivalent level of programming.
> simple fact that shows this is the amount of programmers who have no formal training in engineering or computer science but we’re able to self-teach the concepts. The same cannot be said of mathematics
I would blame it more on the fact that programming is a very useful tool for people outside Computer-Science, it has very direct applications and you can monetize it very easily, so it's very likely that they might want to learn it, however, rarely you see someone deciding to take Calculus just for the sake of it if they never bothered with it in College.
Overall I agree with you, but personally I find math more difficult, most people probably do but I don't think it's inherently more difficult, it's just that people are less used to study it.