I was also wondering recently why illiteracy carries social stigma, while innumeracy does not. The article asserts everyone just considers math ability to be innate, but I'm not sure this is a good explanation.
Our daily life seems to rely on literacy much more than it relies on numeracy. You can only know how to add and subtract (mostly money) and you will still function all right in your daily life and a great variety of jobs. Not so much if you have difficulty reading. This seems to be slowly changing in the future, but if you consider the learning curve, mathematics has a much steeper one compared to reading and writing well.
Does anyone have a better explanation?