Personally, I'd say this is a good sign: if your conference makes people like ESR and the other posters in the linked thread feel unwelcome and uncomfortable showing up and bringing the kind of attitudes and behavior that make other people feel unwelcome, then keep it up, you're on the right track.
I don't now much about this particular conf or the people, so maybe I'm misjudging you. But in general I believe many people are offended far to easily and we give far too much weight to them. Never being offended is not a human right.
Here's a hint: I've known lots of trans and nonconforming people who use whatever unisex bathroom they feel like when they're at an event like this, and nobody cared, because these communities accept lots of "non traditional" types. That's the whole point. Let your personal flag fly.
If you come back with "Oh but that's a micro aggression against trans people"... Welcome to the problem. Just punch me in the face, I'm a Nazi. QED.