This is highly cultural. And I'd be surprised if, in the US, men "fear" gay men the way you're making out. But as OP clearly stated, there are countries where mixed sex naked saunas/spas are a thing and ogling or other kinds of harassment are not a big issue.