[For instance, someone could overhear it out of context and think you're being inappropriate. That may not amount to anything in isolation, but then they might start selectively watching for problematic things you say or do based on their initial mistaken impression, and if you look for something hard enough you're likely to find it.]
Also, I've noticed even when I use the word sex in that context, I'm immediately on guard against making any kind of sexual reference or anything that might have double meaning, and that's just exhausting. I'd rather use gender and save myself the trouble.
GP quoted the wrong definition, but the right definition is there too, also from Merriam-Webster:
2a: SEX sense 1a the feminine gender
If you're needing to use the word sex, it means you're talking about the difference in shape of a person's private parts. The word is exactly as socially acceptable as it seems to be.
If you're trying to compare a bunch of people in a room, like to say you should divide up teams by ___, you probably want gender, not sex. It's medical professionals who usually have cause to care about the sex of a person they know. Unless you're dating someone, and then it is sex sex you're talking about...
Now people are taking "gender identity" and shortening it to be gender and taking exception to people using the synonym as we always have.
Webster's 1913 does appear to regard "gender" as exclusively a grammatical quality, so far as definitions related to this topic go. So the modern usage may not be quite that old.
[EDIT] huh, whaddaya know, Garner's Modern English Usage puts the move to "gender" in places where "sex" would previously have been used (to refer to all or to any part of the whole deal, not just the biological bits) as right around when I first encountered it, at least outside an initially-narrow set of academic disciplines. Go figure. It is newer than I thought, by a long shot. I figured it dated to the 60s or so in academia, mainstream by the 80s and I just hadn't encountered anyone who cared to make the distinction until later.
I kinda halfway try to follow this stuff, and honestly, I'd have trouble defining most of the terms involved in a way that didn't step on someone's toes—and I'd be trying not to! No wonder people are put off by it.
That's 99.9% of it, right there.
So just be honest and try to be polite, that's the best you can ever do. Whether or not you claim that gender is "entirely social" should depend on what you believe, not what you think the safe answer is.
Sex is not a bad word. If someone misinterprets your use of a word, that's their problem.