I think the people this effects the worst are those who don't subscribe to either party, or the idea of a "one-dimensional political spectrum" in general. The dominating political "sides" are full on ideologies at this point, and if you don't worship their sacred cow: you could only ever be the other side in their eyes.

So the price for not falling into the two-party trap is that you have to just keep your mouth shut in all situations, lest you be ousted.

Anecdote: all of my friends who very much aren't falling for this two-party nonsense have had a common experience of being called Trumpers, Leftists, Anti-fa, Rednecks, and every other slur one side throws at the other. I'm not sure if it's a real litmus test or not, but I start to worry now when I've been getting too much of one and not enough of the other.