In the end, we would get 5 to 10 times as many talk submissions from men, accept nearly all of the talks by women (often turning down higher quality talks from men) trying to get diversity numbers and have a much higher cancellation rate form the women for various reasons, especially money. If we were lucky we would get 20-30% women speakers when all of that was done. I've also been told by some diversity activists that, in order to get women to speak and to get higher percentages of women as speakers etc., unlike most of the men, you need to pay for their travel etc., offer free tickets for women attendees, consider offering stipends for women who speak etc..
Others, organizing other conferences have shared similar experiences. It's one thing to fight against blatant sexism which I'm for, but this is turning into something else.