Over time, I realize that no one needs to do anything wrong for unfortunate things to happen. Even though I didn't do anything wrong and didn't mean anyone harm, the unfortunate lack of females at conference still happens. It happens because of historical accidents like home video games that introduced more boys to technology, for example.
So nowadays I try not to interpret complains about diversity as a complaint about me personally. (And I try to remind my feminist friends not to appear like they are complaining about men personally.)
I also no longer get upset about women-only events and organizations. They are free to gather and support one another, just like we have association of business owners, school alumni, even guys named Smith! They are building each other up instead of tearing other groups down, so there's nothing wrong about that.