If you are talking about race or gender than yeah, the "conservative" position is terrible and you would rightly face consequences for your awful beliefs.
If you're talking about telling your coworkers that you voted for Trump, well, I don't think that's something that ought to be taboo but he did explicitly embrace the terrible beliefs I mentioned so I think that explicitly renouncing them wouldn't be a bad idea.