Math has typically been taught devoid of any context or relevance to our everyday lives. Even in college, you just mindlessly plug and chug formulas to get by. Some students don't see the use in it, beyond perhaps the parts needed for basic financial literacy (which many do not have, either).
Better and more engaging and relevant and effective ways to teach math have already been developed, such as Realistic Mathematics Education.
But people actually say things like "I can't do math" in regards to all forms of literacy, I believe. It depends what standards you are mentally comparing yourself to, I guess, such as professional vs. social standards.
"I'm not good with money, or I'm not good with the business stuff" - financial literacy
"I can't write" - as in, I can't write books or novels or easily do other professional writing tasks. Again, there are pedagogical techniques that can help students write more, write better, and have more interest and confidence in writing.
"I don't read" - most folks nowadays don't really read books or novels anymore, what with TV, movies, and the Internet.
"I don't know computers" - how often do we hear that - that's computer literacy. I hear it less and less though nowadays.
To tell you the truth, maybe it will be nice when one day people feel forced to admit "I can't code." Because that would mean that programming and computational literacy is something taught in most schools.