One pillar is alignment of values, and therefore intent. The other pillar is competence.
These are the same issues faced by AI development, as well as representative government, or anything regulating a dynamic with competing elements or agents.
Yet our plurality voting system would be insufficient even to keep a car on the road and driving within the speed requirements. If only the founding fathers had recognized the need to have more information included in ballots so that negative campaigning wasn't as effective if not more effective than positive.
If we voted with {+1, +0.5, -0.5, 0, 0, 0...} weights, without duplication of non-zero values, the smartest, most constructive candidates would have a better chance. Each district would have its own blend of 3-4 viable parties, and the nation would be all the healthier for it. (Side note: Yes, this is still one person one vote--you could imagine voting with a single checkbox for a single permutation of all possible assignments of the scores, as an intermediate form.)
Back to your point, though: Yes, incompetence and malice can have the same effect in the short term. The long term is what determines the difference, both in effect and our responses to it.