I'm just throwing out my own intuition here, but I think that firstly, there's an implication that you have a penis. You don't? Odd one out.
Secondly, if a woman were to say that verbally to a man, there would be a flirtatious connotation, which would lead women to feel uncomfortable wearing a shirt saying the same and hanging around with a load of strangers.