What they're saying is: "We reached out to the best, and it turns out that the ones who accepted our offer were white males. People complained about that; but we don't want to go out of our way to find a non white male just to tick a box".
See, they should have gone out of their way to do this. Not just to tick a box, but because ticking that box is a kind of due diligence towards building a more inclusive community. Unless they're ready to claim that there's no interesting non-white, non-male speakers able or willing to speak at their conference, then the failure is theirs to make any effort at all. But if they did make the effort and failed to attract someone, then they can say "we actually tried, and couldn't do it." The effort need be no more than to send a bunch of emails, and if that fails, it demonstrates a deeper problem in their community than they can fix.
Why don't they quantify who 'the best' that they reached out to where?
Here's my point: frequently, people say 'the best people just happen to be white males' or 'we can only hire white males'. In reality, this is often because there's been no effort made to reach out to other segments of society.
The tokenism excuse is then deployed - but that isn't the point. No-one likes tokenism. The argument we're making is that there are talented people who get overlooked or are made to feel excluded (often accidentally) because they aren't white men.
I find it incredibly offensive when people deploy the tokenism argument against this criticism because the implicit claim is that there aren't enough talented members of minority groups. This isn't the case.
Moreover, it's part of the continuous passing the buck you see in sectors like tech. Businesses say they don't hire more women because there aren't many women with CompSci degrees. CompSci programs say they don't get more women because women aren't interested. And yet companies and faculties are still willing to have woman-unfriendly cultures, because women don't want to work there.
To be specific to conferences - if every conference says "we reached out to good speakers but there weren't many women" then there never will be - because women won't get their shot.
To be fair, I don't know the specifics. Maybe this conference did make efforts. But given that they haven't quantified what effort they did make, their defence is extremely weak.
This isn't clear cut, or black and white, it's hovering somewhere in the grey in between, in all honesty, if you were a minority would you rather be told "we were told we had to get a middle aged, homosexual black woman because we have a bunch of racial and sexual checkboxes we need filling or we can't go ahead"?
In this case the only options left to the people running the event were cave under the ridiculous pressure put towards them or press ahead and get a load of bad publicity and branded as racist, sexist ass holes instead.
It's all well and good to promote equality but when it falls on the other shoe and it's doing it for doing it's sake instead of actually trying to make a positive effect, what you end up with is everyone feeling uncomfortable and putting them in such a poor position that this event (that I'm sure as hell people of all ages, races and genders would have attended) got cancelled is downright wrong and actually shows one of the nastier sides of promoting equality, which is when it digresses in to mindless anarchical bitchyness.
Either way, being offended isn't an argument.
It's a small, but important, difference.
It's a perfectly reasonable argument - but it is frequently made inappropriately in an attempt to counter criticism that it doesn't really offer a valuable defence against. It's an attempt to shift the terms of the debate. I don't like it.