This is actually something I think will go away once we manage to overcome sexism in society. There is really no reason for it at all.
My subjective impression of you and your "anti-sexism" is that you are an ideological fanatic. Fortunately, it's not likely that you and your kind will ever "overcome sexism in society" and it's likely that public restrooms will remain segregated by gender.
Your understanding of the differences between the sexes seems based more on ideological, abstract categories than practical, biologically based considerations, which is often the case with ideological and religious fanaticism.
In my opinion, your "anti-sexism" is just rehashed, 20th century cultural Marxism that is fortunately becoming less popular. I hope we can all progress beyond such failed, politically and religiously motivated trends.