I don't think a declaration that the existence of all-male panels is in an of itself a bad thing that needs to be wiped out (as opposed to a symptom of systemic sexism/pervasive sexist attitudes). Immediately jumping to a belligerent, absolutist position on the matter that won't be well received by anyone except people looking for good-person credits by taking a stand isn't helping either.
The underlying idea of Affirmative Action is sound: instead of just taking people at their word when they say they aren't bigoted (they probably aren't, inertia is sufficient), expect them to show some sign of it: Not just passively waiting for someone who isn't a white dude to wander into their social circle, but have an actual process.
For example, you look outside your Rolodex of usual invitees for some qualified new blood. If you don't find any, have your white dudes conference. It's the existence of a credible process for an outsider to get in that establishes your good faith: not some magic ratio or token representation, and not empty promises that you're totally not one of those bad sexist/racist/whatever people.
The bad news is that before anyone gets anywhere along that path, the usual brigade of people looking for any excuse to exert coercive authority over other people jumps in and starts perverting it into the usual garbage of "Not quotas, by which we mean quotas", leading to "oh you only invited $group to fill the quota" or just even "Fund our group for an indulgence". Ugh.