It's a matter of how much we all have to agree to.

Agreeing to contract law is fine, for example.

It's possible to go too far. If you're going to use force to get people to do things they don't want to, you might be going too far. Naturally, there are such things that aren't tyranny: taxes for example. Even with taxes there are levels of taxation that are tyrannical. For example, a 100% property tax would be tyrannical.

People are going to disagree about where the boundary between tyranny and not-tyranny lies. If you have a very sizeable minority saying "this is tyranny", well, it probably is (but doesn't have to be).