In my experience, it generally becomes clear to everyone if someone is really not working out, and it's already been the subject of hushed conversation.
If you need to "whip up votes", this will say more about you than the person you think isn't doing a good job.
Borderline cases are trickier, but they are for managers too.