What they're saying is: "We reached out to the best, and it turns out that the ones who accepted our offer were white males. People complained about that; but we don't want to go out of our way to find a non white male just to tick a box".
See, they should have gone out of their way to do this. Not just to tick a box, but because ticking that box is a kind of due diligence towards building a more inclusive community. Unless they're ready to claim that there's no interesting non-white, non-male speakers able or willing to speak at their conference, then the failure is theirs to make any effort at all. But if they did make the effort and failed to attract someone, then they can say "we actually tried, and couldn't do it." The effort need be no more than to send a bunch of emails, and if that fails, it demonstrates a deeper problem in their community than they can fix.
Why don't they quantify who 'the best' that they reached out to where?
Here's my point: frequently, people say 'the best people just happen to be white males' or 'we can only hire white males'. In reality, this is often because there's been no effort made to reach out to other segments of society.
The tokenism excuse is then deployed - but that isn't the point. No-one likes tokenism. The argument we're making is that there are talented people who get overlooked or are made to feel excluded (often accidentally) because they aren't white men.
I find it incredibly offensive when people deploy the tokenism argument against this criticism because the implicit claim is that there aren't enough talented members of minority groups. This isn't the case.
Moreover, it's part of the continuous passing the buck you see in sectors like tech. Businesses say they don't hire more women because there aren't many women with CompSci degrees. CompSci programs say they don't get more women because women aren't interested. And yet companies and faculties are still willing to have woman-unfriendly cultures, because women don't want to work there.
To be specific to conferences - if every conference says "we reached out to good speakers but there weren't many women" then there never will be - because women won't get their shot.
To be fair, I don't know the specifics. Maybe this conference did make efforts. But given that they haven't quantified what effort they did make, their defence is extremely weak.