I mean, I'm definitely a naive progressive (and probably hold some internalized bigotry that I'm unaware of), but I was against the SAT/ACT requirements sort of for the opposite reason than what you described.
I half-assed my entire way through high school, but studied for about a month for the ACT and did extremely well (perfect in all categories except Math, which I got a 32 in). I didn't get into MIT (I never applied) but I did get into a few other relatively well-regarded universities (Auburn, NYU) despite my awful grades, almost exclusively riding off the strength of my ACT scores, and ended up going to Florida State (since it was cheaper than the other two I listed). I dropped out after 2 years (nearly flunking out due to low grades).
To me, this showed that the tests are not an accurate measurement of how successful someone will be at college, but instead just how well someone can prepare for a specific test. If that's the case, why add the extra cost, both time and money-wise? If a mediocre student can just cram for a few weeks and do well on the test, then it seems to me that it's not a great test.
I acknowledge that this is pure anecdata, but that was my perspective, not "minorities can't be expected to do well on anything objective" and not "I don't want Yale to be 2/3 asian."