My theory is that math anxiety is really anxiety about a cold assessment.

In other subjects you can rationalize to yourself in various ways: the teacher doesn't like me, or I got unlucky and they only asked the history questions I didn't know.

But with math, no rationalization is possible. There's no hope the teacher will go easy on you, or be happy that you got the gist of the solution.

Failure in math is often (but not always) a sign that education has failed in general. Teachers can be lazy or too nice and give good grades in art or history or reading to any student. But when the standardized math test comes around, there's no hiding from it (teacher or student).