Yes, you don't need to be part of the 50%, but that doesn't get you away from that effect of the 50% either. There are causes in both major parties that someone is likely to find themselves on the receiving end of some law or regulation that they don't like. So you could still end up opposing that 50% if you're lumping them all in with the politician on that side.
"If you vote for a candidate knowing that they will use the power you give them in ways which are harmful to others, you're contributing to the problem."
Can you name any candidate that wouldn't harm any group? It seems they all have ideas about what laws would make things better, and those laws tend to involve control of people's actions or inactions.