[For instance, someone could overhear it out of context and think you're being inappropriate. That may not amount to anything in isolation, but then they might start selectively watching for problematic things you say or do based on their initial mistaken impression, and if you look for something hard enough you're likely to find it.]
Also, I've noticed even when I use the word sex in that context, I'm immediately on guard against making any kind of sexual reference or anything that might have double meaning, and that's just exhausting. I'd rather use gender and save myself the trouble.
GP quoted the wrong definition, but the right definition is there too, also from Merriam-Webster:
2a: SEX sense 1a the feminine gender