That is but one possible situation. Read the link. There are other things like adrenal hyperplasia which are really common and results in developing secondary sexual characteristics of both genders.
Is being a man a quirk or a disease of yours or in the core of your identity?