Why isn't it? Those agreements don't restrict or infringe on the freedom of others to form different agreements, or avoid forming them.
> What about colluding on prices?
Same exact situation. Price agreements don't restrict the freedom of other parties to set prices contrary to that agreement.
> Aren't these all the same infringement?
Without getting into specifics... Yes.
> The truth is sometimes society has to take away the rights of the few for the sake of the many.
I truly commend you for your honesty about what these laws are.
> The fact is rules like this almost certainly make our economy more productive as a whole.
I don't see where that's been demonstrated here. They certainly aren't productive for the firms who are punished for, or prevented from, forming such agreements. I would argue that the fact that firms do form such agreements when free to do so, means that they are productive for the economy as a whole by definition.