Your last point is well-taken. It may even be the case these days that the majority of people who hold views similar to yours think the way you do. But I think we need to take into account the uncertainty that people in these situations face. It's hard to know what the other person is thinking -- and if all you know is that person's opinion, it's hard to know what's going on when you're not looking. Bigotry is often dressed up in talk and behavior that seems polite, even respectful at first glance, but that is ultimately materially harmful.
For example, I think a lot of LGBTQ people from religious backgrounds have had an experience of being told that their family or community will "love the sinner, but hate the sin" -- and then subsequently being subjected to unfair and harmful treatment (bullying, psychological abuse, ostracism, being disowned).