Your assumption seems to be that if no-one physically stops these men, they can and will do what they like, with impunity, and that there's no chance of them simply respecting women's boundaries. Is that what you mean?
That seems like a reliable assumption to me. Your assumptions seems to be that if you make a "rule" concerning who can go into which bathroom based on a hidden characteristic, no one will take it upon themselves to enforce this rule, and thus to make frequently wrong guesses regarding this hidden characteristic. That seems like a poor assumption.
>and that there's no chance of them simply respecting women's boundaries
Different people have different boundaries. Different people have different standards as for who should be sorted into which bathrooms. One person's arbitrary standards cannot dominate everyone else's lives. Instead, rules must be made based on reason and fairness, as I demonstrated above.