Because it makes it sound like you're more interested in hitting on that person than interacting with them professionally. And unless you're inclined to call everyone, regardless of gender, "sexy," it shows that you can't work with members of the opposite gender without putting a lot of emphasis on how bangable they are. Being called "sexy" tends to emphasize that you're a nicely arranged sack of meat, not a competent professional, and can make a woman (or man) think that your genitals are going to get in the way of a respectful working relationship. Not only that, but sexy people tend to be perceived as less competent. Just think of the "blonde bimbo" and "unattractive female in software engineering" stereotypes. Think about how dumb jocks get the high school cheerleader, and skinny, ugly nerds don't.
Now, I'm sure some men (or women) can successfully use that word to describe their colleagues and still work productively with them, but quite a few men can't. For an egregious example, see this profile about Sheryl Sandberg: http://www.newyorker.com/reporting/2011/07/11/110711fa_fact_...
"Dina Kaplan, the co-founder of Blip.tv, says that when she met with angel investors to raise funds she dressed nicely, and in a meeting with a potential funder he told her, “Here’s what we do, Dina. We’re going to spend half the meeting with you pitching me, and half the meeting with me hitting on you!”
“I felt nauseous,” she says. “I tried to laugh it off. I asked, ‘Of all the things you’re working on, what most excites you?’ He said, ‘Seeing you naked tonight.’"
When you've got men like that out there describing their female coworkers as sexy, you're going to have to work pretty hard to convince women not to lump you in with them. It's unfair, but that's how stereotypes work.