Yeah it's more about people interacting in a voluntary manner. The first and probably last rule of libertarian philosophy is that it's not OK to initiate aggression against others. So two individuals entering into a contract is OK but one person threatening another with violence is not.

The fact that the state/government has a monopoly on violence means that your interactions with them aren't necessarily voluntary. Many people don't have a problem with paying for roads, or some semblance of a military, or schools or whatnot.

But the very nature of a state is that if you don't do what it tells you to, you have no recourse. Someone might not put a gun to your head but you have to know that there's no negotiation.