You can always click 'unvote' in the detail line.