It's true rainbow activist types use sexual orientations and minorities as shields when their politics get criticized, as already seen above.
If people say "listen to gay people" they hardly mean say, UK writer Douglas Murray or the activist group Gays Against Groomers.
If people say to listen to black people, I'm quite sure they don't mean economist Thomas Sowell.
If they say transpeople, they definitely don't mean Scott Newgent of "What is a Woman?" fame.
All these people have the correct identity groups, but they're no-nos because they have the wrong politics. The DEI people clamoring for better representation definitely don't want offices staffed with Sowells and Murrays. Because the identity is just a shield, and the politics is the point, as eg. Rep. Ayanna Pressley put when she said they need "more black faces willing to be a black voice" - again, Sowell certainly is black and has a voice, but Pressley meant having the correct politics to count as representative.
Wanting less DEI is wanting less of a certain kind of politics, not wanting less people of whatever sexual orientation or skin colour. Because the politics is the point.