I agree with this, fully and unequivocally. I have to wonder if "there are ladies present" might also be construed as sexist. Maybe we just shouldn't use toilet humor in situations where we're not absolutely sure everyone will find it appropriate.
Warning: offensive, but to demonstrate a point...
I once interviewed a guy who decided to wear a t-shirt to the interview that read "Thousands of my potential children died on your daughter's face last night". There were no women there. I was not really even offended (toilet humor is really hard not to giggle at, if for no other reason than I'm still 15 in my head somewhere). I was, however, careful not to let on that I noticed it at all. On the way out, he asked for a tour (he thought he was getting the job). When another coworker commented on his shirt he said "Yeah, I got another one that says 'Swallow it or it goes in your eye'." It was a 5 person company at the time, and none of them were female. Still, he wasn't getting the job; anyone with that much disregard (dare I say arrogance) for other's feelings isn't getting the job.
It's not about having men or ladies or kids present; it's about having a mutual respect for others, and being conservative about what setting they think toilet humor (or any humor) is appropriate.