Exactly! And that's why changes like this are so important. I, like you, don't want to be pressured into referring to someone by a gender identity which does not match their reality.
The transgender "agenda" isn't forcing anyone to do anything, it's about having basic respect for the identity of those you interact with.
You're free to reject "other people's illusions", but there's a good chance you'll just come across as insensitive and a bit of a jerk.
Your personal relationship to gender is very simple. Yay, good for you. But there are people whose relationships are more complex. I promise that they have thought about gender 100x more than you have, and it is you that have the illusions here.
How do I know? Well, I started out with those same illusions and then actually talked to people who changed gender or who just don't fit well in a gender-binary framework. I learned a lot. If you're willing to set aside your arrogance on this, you could do the same.
The first half involves judging people by appearance; unless he's running DNA tests, ultrasounds, and fondling everybody he meets before he uses a pronoun, then he his judging at least partly by how they present. That's gender.
The latter half, how he treats people based on his gender-presentation-mediated perceptions of sex, is pure gender, because there we're entirely out of the realm of biology and clearly into the social side of gender roles.
I understand why he (and apparently you) would like it to be simpler. It's more convenient for him if everybody goes around hiding the complexity for him so he doesn't have to have a more accurate view of the world. But it's bunk. Pretending that it's "just" sex he's talking about is part of the way he hides, presumably also from himself, that it's bunk.
So, even from the "biology is the only real thing that legitimate justifies gender identity" perspective -- a perspective that I personally find ludicrous but which is nonetheless common among people who whine about a "transgender agenda" and assert that gender identity is somehow invalid when it doesn't line up with biology -- even from that narrow-minded perspective transgender people (or a very significant share of them, at any rate) have a legitimate claim to a distinct gender identity.
The illusions about sex and gender are yours, not theirs.
For example, what gender would you assign to someone who was born with both sets of genitalia, and had the choice made by their parents to remove the vagina? She may choose to identify as female because that's how her brain (and possibly breasts) evolved, regardless of the choice her parents made for her as a baby.
Sex is biological, gender is a label people choose to identify with. How could it be an illusion? Would you prefer to inspect their genitalia before using pronouns? Men could always label themselves as female if they want anyway, now they simply have more options.
Do you also refuse to use nicknames because they're different from their legal names? What about Internet handles?
Or do you not want to be pressured into describing things accurately?