Co-workers routinely ask personal questions, expect small talk about details of their lives out of work. I find this uncomfortable to, but am forced to participate so I don't become seen as 'not a team player'.
To be absolutely clear, I am not condoning the kind of behaviour you have experienced as something I consider to be ok. By the sound of it, it's certainly not.
However, my point is that almost always its a matter of perspective, its not just sexual related topics that make people uncomfortable. For many of the individuals you have spoken about, their personal understanding of the situation could legitimately be that it was ok.
Quite frankly, I just don't know what can be done to solve a problem so complex as individuals understanding matters in different ways.