You're mostly right. Unfortunately, there are enough extremists who want actual total "equality" (whatever that means - should beautiful people get scarred to be more equal?) that it's an easy straw-man to use for the regressive types.
Are there? Where are they and how much influence do they have? As far as I can tell they are extremely rare and have little to no power. In contrast, the number of people who are market maximalists in actual positions of power is significant.
Interested to see any evidence that my world view is wrong.
The other turns that around and says no we don't because "structural racism/sexism/classism/etc".
So, they argue against each-other because one sees the current inequality of outcome as a consequence of some sort of "biased" fairness in opportunity. And since the other side disputes the existence of that bias, they think that what's being advocated is pure equality, irrespective of any distribution of skills/talents/etc.
It's an impasse, and we'll never get passed it until we start agreeing and quantifying the possibility that there are differences between sexes/races/cultures/ages. Only then can we correct for any supposed structural biases. It's a touchy topic, and one only need look at the type of controversy The Bell Curve book caused to understand the incredible difficulty that we face trying to conclusively resolve this scientifically.
If you look at history of mankind, you will see that at any stage the existing differences seemed insurmountable and later nevertheless changed. You can look anywhere from the Roman empire (i.e., essentially a fascist militaristic society for a long time), over women's rights, slavery, colonialization etc., it's always the same pattern. The status quo has never defined how a desirable society should look like. As a typical example, many women used to agree that it's pointless for them to vote, since they don't know anything about politics. Maybe these were even in the majority at some time. Nevertheless they've changed their mind.
To cut a long story short, history is full of alleged anthropological constants that were used to justify inequality and all of them turned out to be false and nonexistent. Heck, landowners used to say about peasants that they are just too stupid to own their own land and would immediately gamble it away...
Quite a lot, one was a presidential candidate complaining about the wage gap, which is an argument for total equality, not equal opportunity.