If you can't discuss last night's episode of GoT without being seen as a sexist jerk, you might be a sexist jerk. The fact you keep suggesting you have to "put on kid gloves" suggests you might be a sexist jerk.
I agree there is a place and time for everything. However, true friends are rarely made in polite company. I prefer to work with friends.
It's a subtle thing, really. One person can make remarks about women and shoes and it's funny because we know that's not his real opinion, another does it and it's offensive because he thinks it's true. It really is about what your real opinions and intentions are.
Perhaps step one should be "find out their real opinion and intention". But too often people just go straight to "thermo-nuclear melt down".