Hum. Do you always check a user's profile before downvoting if you don't remember seeing him before? I can't imagine myself doing that.

Or are you using some sort of plugin that gives you more info inline?