I see his larger point, but the conflation of "understanding math" and "being able to do simple arithmetic" is something of a pet peeve. The latter may be a useful life skill, but is not very closely related to understanding mathematics. In fact it's not only non-technical office workers who can be proud of not being able to do it: if you walk into any math department you will find accomplished mathematicians almost gleefully recounting how bad they are at doing simple arithmetic. They can prove things about topological spaces but can't balance a checkbook!

That makes the comparison to literacy more complex. Clearly mathematicians think mathematics is important, or they wouldn't be studying it. But many of them consider ability to do arithmetic, as a skill, essentially irrelevant, to the point where they don't even learn it themselves. Then a question might be: what is the relevant literacy-like skill? I think it comes closer to logic and analytical thinking than specifically mathematics, although understanding of statistical arguments fits into that category.