I feel like anyone should be anxious if they don't understand math. How does one get through life without understanding math? Probably by getting had a lot.
I saw this, teaching college math for one semester long ago, and also in talking with adults about their experience in high school and college math.
As a consequence, a person may be completely functional at basic math, such as balancing their checkbook, while simultaneously believing that they don't understand math.