"It basically says that it is important that teachers are not racist and be aware of their implicit biases. That seems pretty reasonable and non-controversial."
That is definitely not what they are saying in this document.
They are saying that math and the way that is taught effectively around the world is a form of 'White Supremacy'.
If you don't think that isn't controversial, and probably racist, then what is?
You're missing a little bit the underlying philosophy here, when they say 'stop racist teaching' they don't really mean racism in the common sense.
They are implying that math, in the manner that we view it is in and of itself systematically racist.
Detractors are not randomly throwing around the term 'White Supremacy' etc. - here is the very first paragraph of the note:
"to dismantling white supremacy in math classrooms by making visi- ble the toxic characteristics of white supremacy culture"
They are directly implying that a 'focus on the correct answer' etc. is 'White Supremacy Culture' - which is not only ridiculous, but arguably very racist in and of itself.
Consider that there is absolutely no evidence that math instruction is racist. None whatsoever - in fact - almost none of their wild assertions are founded in any objective analysis.
They are ideological: it's a fundamental redefinition of the terms 'racism' etc. towards their view that minorities are systematically oppressed de-facto by the expression of the majority culture.
"White supremacy culture shows up in math classrooms when... Grading practices center what students don't understand rather than what they do understand."
"White supremacy culture shows up in math classrooms when... Students are required to “show their work” in standardized, prescribed ways."
These are interesting thoughts on teaching but 1) there's no evidence presented that teachers are currently engaged in these practices 2) there's no basis for correlating these teaching methods with 'White Supremacy' and most importantly 3) no evidence that the proscribed methods are better in an way.
That people are trying create better learning conditions for everyone is positive - but don't conflate solutions, especially those based on arbitrary ideological objectives - are necessary valuable, in many cases, they can be destructive.