As a consequence of respect for their personal identity as a member of society (and NOT fixating on their sex organs, which is frankly creepy), most people will refer to a trans* person as the gender they identify as, not the sex they were born with.
Or, if uncertain (because of genderfluidity and new pronouns like Xe/Xir/whatever), just using "singular they" and moving on with your life.
It's a polite society thing.
For me, I tend to assign gender by sex. I don't want to disrespect myself by saying something I don't believe. On the other hand I don't want to disrespect others either. So for now my approach is to sidestep the issue and refer to all trans people with "they".
We are not talking about cross-dressers. We are talking about transgender individuals. You wouldn't even be able to notice them standing in front of you once their transition is complete.
I don't understand that part. I think there are some concepts that are confusing you. All of the transgender individuals that I have known from before their transition pass as their new gender without any issues. The only way anybody would know if you told them.
Story time:
You know how men always like to boast and compare their masculinity while drinking? Well, one time we were out drinking with friends. A man from my high-school (who was a girl last I saw him) was with us. He is quite manly, has a beard and looks like what you would expect a fashionable young man in his mid twenties to look like.
After a few beer one of the guys turns toward the man in question and, without having a clue that this is a transgender individual and tells him: "We know how it is with you lucky guys who hit the genetic lottery and are so manly. With a beard like that I bet all the girls must be throwing themselves at you. I must be half the man you are." ... My jaw dropped.
It was a ridiculous situation to witness but it shows just how easy it is to be next to someone who has gone through a complete transition without having a clue. If you live in a city, I'm sure that you have called at least one or two transgender individuals by their new pronoun without being aware of it.
I do the same, but also for cis people. It makes life simpler.
That's an opinion.
Gender is defined as: the state of being male or female (typically used with reference to social and cultural differences rather than biological ones).
For most people, your gender and your sex are congruent. For others, it isn't.
Even if you don't agree: Why do you care?
Definitions are also not fact, by definition.
There are plenty of valid arguments for calling trans people the gender they want to be. This isn't one since it isn't going to convince anyone who doesn't already agree with you.
Edit: The why do you care part is a valid argument, was addressing the rest of your comment