My experience is that it's no big deal if everyone else is doing it. On the other hand, it's rare to find a critical mass of men who don't care about being seen naked unless your gym has a bunch of men over 50.