That same contract is also in force in the absence of a government monopoly on violence, with the difference that the "and die" part gets implemented far more frequently.
That's why anarchists (of any stripe - left or right) shouldn't be trusted, since what they are really aiming for is to destroy democratic institutions and replace them with an authoritarianism that enforces only their preferences.
Only when the government is authoritarian. You can't vote out a dictator, king, warlord, or mob boss.
In my view the govt necessarily has a monopoly on the use of force which is unavoidable. The important part of the "social contract" is when is it valid for the govt to use that force. The US Constitution originally laid out when the use of that force was moral. Using force to extract tax payment from property owners is not on that list and turns the govt into criminals operating under the color of the law which is terrible situation.