For the market to be manipulated, you would need something that lives outside of market rules, namely politicians who can use violent threats to shape the market to benefit them and their friends.
In both cases, if we get rid of a government and we move law making and law enforcement in the hands of the free market, we end up with a system which is the closest representation of what citizens want. It may not fair or just according to someone's ideals, but it would be a close reflection of what people wants and it would remove the form of violence which is most commonly perpetrated in the world: state initiated violence.
All systems of economics are also systems of power. If the only system of power is the economic one than your economy is your de facto government.
If a structure for “governmental“ power is identical to the economic structure you have zero ability to fix abusive power imbalances when they eventually arise. It’s the same reason it’s good to have many checks and balances within government itself. Every structure has flaws, and any singular structure of power will have it’s flaws exploited unchecked.
In contrast, anarchists are usually opposed to having any state whatsoever. The ideology tends towards being far more extreme and uncompromising.