Warning: offensive, but to demonstrate a point...
I once interviewed a guy who decided to wear a t-shirt to the interview that read "Thousands of my potential children died on your daughter's face last night". There were no women there. I was not really even offended (toilet humor is really hard not to giggle at, if for no other reason than I'm still 15 in my head somewhere). I was, however, careful not to let on that I noticed it at all. On the way out, he asked for a tour (he thought he was getting the job). When another coworker commented on his shirt he said "Yeah, I got another one that says 'Swallow it or it goes in your eye'." It was a 5 person company at the time, and none of them were female. Still, he wasn't getting the job; anyone with that much disregard (dare I say arrogance) for other's feelings isn't getting the job.
It's not about having men or ladies or kids present; it's about having a mutual respect for others, and being conservative about what setting they think toilet humor (or any humor) is appropriate.