In the free market cartels break down very fast, because one member eventually decides to defect or because another powerful player comes to the market and realizes he can make a lot of money by providing customers with a better and cheaper service than the cartel members.
Generally, if most people agree they don't want rulers, there will be no rulers. The only reason why governments keep their power is because the managed to convince people that the alternative is a complete chaos and destruction, while in reality it is only governments who create chaos and destruction on an unimaginable scale. Even the most oppressive government can't go for a long time using just force, it has to lie and convince people. If it can't, it is brought down.