It seems to me that this argument essentially relies on everyone being an anarchist. This is not the case for systems we have observed in reality - totalitarian regimes can function even if significant parts of population don't want to be in a totalitarian regime; democracies can function (perhaps poorly, but function) even if significant parts of population don't agree with core principles of democracy; oligarchies have functioned as well without much of the society not acting in an oligarchic manner.
Does the same apply to your vision of anarchy? Is your proposed method of enforcing the social contract stable in the face of adversity (a Nash equilibrium of sorts)? I.e. for a non-anarchist participant, is it really in their own best selfish interest to participate in that enforcement instead of defecting, and not refusing to deal with people who don't respect the anarchist social contract because it's good for them at that point of time? In an anarchy, everyone has that option as well, and if the boycott isn't sufficiently strong (which IMHO requires it to be organized, monitored and enforced to work in any community much larger than Dunbar's number), then people who do not respect that social contract do not find themselves ostracized and powerless; especially since they'll be dealing between themselves if the anarchist community does not. Are there sufficiently strong incentives for people to actually 'keep to the plan' and fully ostracize the others? There will be some misdeeds done by well-loved, respected members of the community against someone that's not liked - will those misdeeds be punished at all? It's not sufficient to try to ostracize them if there's enough critical mass (which is quite small) for them to be somewhat self-sufficient; there certainly will be both individual maniacs and random gangs and extended family clans that will laugh at such ostracism, in which case you need to be able to reliably organize enough violence to physically prevent them from breaking the social contract and getting away with it; and you need to be able to get everyone to agree that this person or group has broken the social contract, instead of you just wanting to remove a competitor.
My core point is probably the following:
1. there would inevitably be certain sub-groups run by decidedly non-anarchistic principles, obtaining strong cohesion by whatever means. Cliques, family units/clans, groups of friends, etc - the concept and benefits of cooperation can't be eliminated.
2. at least some of these groups (or their leaders) would have a way to capture more resources and power by easily taking them from any disorganized groups or disjoint individuals, so either:
3a. the surrounding anarchists would be forced to cooperate in a sufficiently organized manner, to become a larger and more organized group than the local strongman/warlord/whatever, and enforce that social contract on them; and be prepared to do it permanently, as otherwise the threat would re-appear - in which case that organizational structure and its decisionmaking process, whatever it may be, is a de-facto non-anarchistic ruler of that area.
3b. the surrounding anarchists do not manage to cooperate in a way that results in them being the largest/most organized group in the area, in which case, their communities are eventually 'eaten up' by the larger non-anarchistic groups/organizations that force their will on them, and eliminate anarchy in that area.
Can you imagine (and describe) how would an anarchistic community operate sustainably next to a miniature equivalent of North Korea, a plantation using slaves, or a religious cult that's eager to push their rules onto everybody they can? How would many village-sized anarchistic communities cooperate to unify against a common threat that one or two villages can't handle on their own? For a social system to be possible, it has to be somewhat stable, it has to be able to overcome competing systems (both from within and from outside); both dictatorships and democracies have shown themselves to be capable of resisting/suppressing at least a certain size of revolution attempts; if a system can be overthrown or overtaken by a reasonably organized band of men, then it will be overthrown, and if such threats force a system to change, then the system will change and the unchanged system can't exist.
All competing systems have processes to eliminate other systems (i.e. enforce their rule) within their 'zone of control' - if anarchy has no plausible process to enforce 'rule of anarchy' i.e. prevent pockets of democracy or dictatorship or theocracy or whatever from rising, thriving and expanding within its zone of control, that's what will happen and the anarchy will cease.