When people accuse someone of being "woke", I believe they do not mean literally that they consider those people to be more awake to social justice issues, that they disagree with fixing injustice, and they want it to continue. Rather, it can be an accusation of performative social virtue, a distaste for sanctimonious platitudes, or a disagreement regarding the source of a particular systemic issue and how to fix it.

Now, you can disagree with this, it is obviously impossible to tell for sure without reading people's minds, and many people will throw out accusations of "wokism" with little to no merit to the point where you lose the will to engage with any of them, but if you do argue against these accusations of "wokism" as endorsements of bigotry, you aren't actually engaging with what people are telling you.

In that sense, the "middle" between "woke" and "bigoted" isn't "slightly bigoted", it's "not woke and not bigoted".