> I mean, where does this political correctness stop
It seems really obvious to me. At work, at a conference, you don't tell racist, sexist, ableist etc jokes. What you do outside work is up to you. (But, if you do it outside work and in public it may complicate things if you have a work related discrimination problem.)
> political correctness
"Political correctness" is often a buzzword used to dismiss people who want to discuss this. Other, better, words exist. "Politeness" is one.