If you speak to a female colleague, is it something you can say to your own mother and be taken as appropriate? Same for if it were a male colleague, are the words appropriate to say to your own father?
If the answer is "No" then don't say it, I wish more companies would enforce this in their culture.
I've run into childish inappropriateness more than anything, phrases not indented to harass, but are extremely disrespectful, not funny, and easily taken the wrong way.
An example: 5 of us, one of them a female, were chatting in a breakroom when a male colleague joked about using a "fleshlight" to relieve stress (I am leaving out details to keep this short). He was straight out of college and didn't have much professional experience. The female in the group became visibly uncomfortable and left. Since the conversation was more about to know our colleague's hobbies outside of work, it was an incredibly awkward turn to the conversation. "Pfff, what's with HER?" the offending person asks as the female walks off, not knowing what he did, so another male in the group asked him if he talks to his mom "like that" about his fleshlight habit, it wasn't until then that he realized he was being inappropriate and he then apologized to everyone. He also sought out the female who left the conversation to apologize to her directly. He learned his lesson, but I've seen examples on BOTH sides of inappropriate talk that can alienate a person who's gender is in the minority of the group.