I don't think "voting permanency == more thought before voting" is true for most people. I was saying it was true in my case and even so, it was a (relatively) long time before I noticed that unvoting was missing and that I still use votes as bookmarks, just less often than before. Plus it seems like an inefficient way to cut down on votes for bad links compared to being able to take back a mistaken vote.
More importantly, although I've reduced my usage of votes as bookmarking, I don't see why that's a good thing. The implied counter example is that if people could unvote, it would be a giant safety net and unleash a tidal wave of spurious votes, with people clicking on stories willy nilly because their actions were reversable. This doesn't sound true - more realistic scenarios are that there'd be a slight bump in votes overall or even a reduction of "bad" votes by allowing people to correct misclicks.
Thinking about my usage patterns, I bet a lot of people realize they can't take back their votes, use votes as bookmarks anyway and are simply annoyed at not being able to trim down their "saved" list by removing outdated/mistaken/irrelevant links. The lack of an undo feature doesn't change most people's actions unless they notice in advance that it's missing because the action they're about to perform is something really important, like paying for something or deleting their writing.
A big chunk of the problem may simply be the verbiage and language of the profile page. I can see how newish people (like myself) misunderstand voting for saving because votes are displayed as "Saved: ##" rather than "Votes: ##" instead. Saving implies bookmarking, as well as unsaving. Being able to "unvote" sounds unnatural, it sounds like what happened to Gore in Florida and nobody would expect to take back their participation in a poll. Whereas "unsaving" doesn't sound weird and is a routine operation.