Seems to me that there are only a couple of different ways somebody could feel about that. The first is to be opposed to the "the smarter you are, the more money you make/better you do in life" meritocratic egalitarianism that these tests support. The second is to support that ideal (whether you agree we're there or not), but insist that tests don't accurately measure intelligence.
Either way, while I can sort of see where the opponents are coming from, I don't see them offering many good alternatives. I don't think it's cosmically "fair" that being born to insanely rich parents leaves you better off in life than being born to unemployed drug addicts, but the only "solution" anybody's ever put forward is massive ongoing wealth redistribution which is demonstrably worse. Similarly, it's not necessarily fair that people who are born smart do better than people who are born stupid, but what do you do? Require as many stupid doctors as smart doctors? Demand that for every smart highway engineer, a stupid one be included as well?
I can also sympathize with people who insist that tests don't accurately measure intelligence - I've always done well on tests, so this system has worked out well for me, but I can see how somebody could be otherwise intelligent but unable to perform on tests. But again... what do you do? Just declare everybody equally intelligent and don't measure anything? Like it or not, tests do measure something that does correlate with that hard-to-quantify trait we call "intelligence" and while there are regrettable false positives as well as false negatives, it's the best we have.
So I wish our tests were harder so there'd be a more fine-grained stratification of actual ability, even if I ended up being closer to the middle than I am now - that would be ok, because a lot of other people would be closer to the middle too! A tests that accurately measured intelligence such that only the most intelligent people in the world could dominate it would necessarily be almost impossible to score perfectly on it, by definition. In the face of such a test, a 50% would be something to brag about and the top scorers would be off curing cancer or something for the benefit of all of us.