This doesn't in any way seem inherent to being transgender, and hence not a reason to avoid them in general? Seems more like a characteristic of some particular people you happen to know, and identity politics or not, generalising that over people who happen to share a noticeable trait with them is what I think we should be careful to avoid anyway.
But then I was recently informed that I spent almost 2 years working with a trans person (m2f) without noticing.
So your mileage might vary.
Is this really a problem? It's a zero cost action.