Are you saying that a penis is what makes you a male? That a vagina is what makes you a female?
What about a man who loses his penis in an accident? Is he no longer a man? Would he still be allowed to identify as a man? Would a woman born without a vagina still be allowed to identify as a woman?