When you say "We are sorry you were offended..." many people will read in to that "...but if it weren't for you being offended there would be nothing wrong with this."
A simple choice of words can make all the difference in the world.
Is this statement even controversial? The only reason someone issues an apology (honest or not) is because one or more others were offended. If nobody is offended, by definition there is nothing wrong with it.
Or rather, if you say something really offensive, like "Women should be allowed to read books, or even leave the house without covering their faces," it's not offensive because you probably aren't saying it in front of a Shia cleric.
If you say something that's intellectually unfashionable, but the people you talk to are unphased, I just don't understand the case for considering that a bad thing. Who is the wronged party? How were they harmed? What's society's interest in redressing this, and what's our mechanism for doing so?
The real question here is would you stand in front of the mirror, with no one around, and say to yourself "The women at this conference are a great perk!" without feeling bad?
It's tempting to think "I have these views about blacks/hispanics/asians/women/etc., but I know how to control my reactions and keep my mouth shut." The problem is that people communicate as much subconsciously as they do consciously.
In your examples, the victim certainly notices and cares; if they get a vote, they're going to vote "immoral." But if you did something "murderous" in absolute privacy--like, say, doing some target shooting--I don't think it would be considered equivalent to murder. Similarly, if you do something "offensive" and nobody is there to get offended, I don't see what the crime is.
Would it upset you if people were saying offensive things quietly to themselves in the privacy of their own homes? How would you propose to deal with that?