The deal is slightly longer: the government gets the right to handle justice, you forgo your right to pursue your personal justice, and you gain the right to not have other peoples' arbitrary sense of justice enforced upon you. And yes, the government bears responsibility for enforcement of personal justice.
If the government abdicates its role, I agree you have the right to enforce your own personal justice. You have not, however, gotten everyone else to sign away their right to not be the target of your vigilante justice.
Put another way, the subject of the "justice" may not feel that the government has abdicated their role and may be content with the way things are going. They have not agreed that the rules have changed, else I strongly suspect they would react differently. If vigilante justice is permissible, responding to vigilante justice with force is also permissible (as is their right to claim what they perceive as "justice").
I also find it a bit spurious to say that the government has abdicated their permission, but the laws are still in effect. If you and the other person have absolved your relationship with the government, the laws no longer apply, and your justification for forcibly removing someone becomes dramatically weaker. You say it's your property and they should leave, they say that was enabled by an unjust socioeconomic system and that reclaiming it was the more just thing to do, and we end up in a very subjective incarnation of justice.