Good points, but at some point the meta must be addressed. People abusing the system will either cause a ton of meta discussions (like this one) or people who don't fit the group think will simply leave.

A way has to be provided to keep accountability and/or defend one's self. On sites like reddit, at least new accounts have the ability to down vote. If you don't fit the group think here you wont even get that chance.