How about an unflag toggle or an undo?
Maybe incidence of this will go down in the future, but the link I'm least likely to click is right next to the one I'm most likely to click, so the mistake is easy.
Although if the intent is to discourage that usage pattern and make people more careful about voting in the first place, then it works, at least in my case. I use votes as bookmarks much more extensively on digg, where I know I can undo them.
Would you write better code if your text editor didn't have undo and you didn't use version control? Probably not.
More importantly, although I've reduced my usage of votes as bookmarking, I don't see why that's a good thing. The implied counter example is that if people could unvote, it would be a giant safety net and unleash a tidal wave of spurious votes, with people clicking on stories willy nilly because their actions were reversable. This doesn't sound true - more realistic scenarios are that there'd be a slight bump in votes overall or even a reduction of "bad" votes by allowing people to correct misclicks.
Thinking about my usage patterns, I bet a lot of people realize they can't take back their votes, use votes as bookmarks anyway and are simply annoyed at not being able to trim down their "saved" list by removing outdated/mistaken/irrelevant links. The lack of an undo feature doesn't change most people's actions unless they notice in advance that it's missing because the action they're about to perform is something really important, like paying for something or deleting their writing.
A big chunk of the problem may simply be the verbiage and language of the profile page. I can see how newish people (like myself) misunderstand voting for saving because votes are displayed as "Saved: ##" rather than "Votes: ##" instead. Saving implies bookmarking, as well as unsaving. Being able to "unvote" sounds unnatural, it sounds like what happened to Gore in Florida and nobody would expect to take back their participation in a poll. Whereas "unsaving" doesn't sound weird and is a routine operation.