By the self-centered logic of the OP, even if a minority group has only 0.1% representation, they should have no events targeting specifically them and their causes -- the 99.9% majority should be able to attend, and presumably contribute to the conversation with no ability of the event's organizers to moderate the dialog or direct the focus to the minority's experiences. Anything less than that is segregation.
It doesn't occur to the OP, apparently, that they are trying to create something that is decidedly different from the ordinary tech experience -- a conversation dominated by awkward-to-arrogant, entitled males who think that their very important opinions need to be heard everywhere and at all times.
I wonder if the OP (presumably male) insists on attending (female) rape survivor meetups, or if he could understand why attendees thereof would not feel totally comfortable with his attendance.