It has apparently crossed several minds, because you're certainly not the first one to suggest it. But as someone who's organized more than one conference, I can say that "the best" is not nearly as easy to define as many people think. Of course there are various kinds of technical quality that we can estimate, and trade off against one another, but one also has to consider the overall balance of the event - for example, we don't want too many talks on the same topics, which may result in otherwise-worthy proposals getting bumped.

Moreover, there is a hugely important signalling role in the selection: it is advertising, "here is our community, here's who we are, here are the problems we work on and the cool techniques we use". So the selection also takes into account things like: "people doing X should get to meet people doing Y", "we think more people should work on Z", and so on. Diversity among the speakers is part of this! Just as surely as you wouldn't want a program of people all from the same company - like it or not, the makeup of the speakers does send a message, and part of the responsibility of organizing an event is deciding what that message will be. It's also communicated by all sorts of other things, including the process for soliciting and choosing submissions, details of language and graphic design, etc. All of these things contribute to a general impression of what the community is like.

The claim of "coincidence" in this instance would be a lot more convincing if the procedures and language were there to back it up. I'm not even talking about affirmative action. I only mean that if you want to claim to be meritocratic, you have to do the work, not just cross your fingers.