it's a known bug. everyone hates it. suggest an appropriate solution. maybe the arrows should straddle the reply link
People can stand by their point or concede that they were wrong somehow.
If your opinion does a 180, it's probably for the better that your - new - opinion doesn't count.
As a side though, if somebody keeps changing opinions on a given topic, it would be somewhat sensible to lower the effect of such votes.
The ability to change opinions is an important difference between rational and overly stubborn people.
Sometimes I can be convinced during a discussion. Then people are often very surprised that I change my opinion at all. Then I wonder why the bother to discuss, if they don't expect me to actually change my opinion.
I think adding a hover effect to the arrow would provide the most benefit per second coding. The following line of CSS can do it without requiring image editing:
a:hover img {opacity: .6;}
Other solutions are better, but this is an easy 80% one.
a[href*=dir\=up] {-webkit-tap-highlight-color: green;}
a[href*=dir\=down] {-webkit-tap-highlight-color: red;}
(edit) Having actually thought about it for a minute, I think a simple "Undo last vote" in the title bar next to the account name would work just fine for reversing accidental votes. And, say, call it unvote for brevity. I really like how clicking on either arrow makes them disappear, this is very much in line with general minimalistic design of the site, so I would rather have it stay.
An alternative is to have the arrows straddle the top line of a comment instead. The down-vote button will be right beside the 'flag' link which may or may not have a desired side effect.
Yet another idea with possibly least side effect is to have them straddle '[x] points'.
Every comment has to have at least 1 line of text, why not make the arrows go from the top of the user line to the bottom of the first comment text line. That is about a 25% increase in size.