Most people are not very even handed when they feel that their rights have been violated, so it tends to lead to escalation, and tribal/gang style violence, tit-for-tat.
It's much better IMO for a society to deal with disputes by legal means, not by stalking, harassing and killing each other, eg. if I shoot you in self defence, then that's 1 person's worth of productivity lost to the world, similarly if I harass you and threaten you at your home address (0.5 x PP?).
And yes, if you consistently violate other people's rights, or act with poor judgement, then you should lose rights. For example, if you're in the US and threaten people with a gun, you should lose the right to carry a gun, or the right to not be searched.