There is always some form of governance occurring, depending on who thinks they are in charge, even if it's not a state. You seem to be advocating for vigilante justice as an enforcement method, which definitely has downsides, just like everything else. Pure freedom is not beautifully perfect and without flaw.