If it's the actual physical agreement that's the problem - the system is working as intended.
But if we are looking to prevent the negative outcomes associated with price fixing and collusion, our system is failing us.
They are never going to find proof of conspiracy. The people involved covered their tracks, and doing so is trivial. So the best we can do is punish the appearance of collusion. And if the goal is to actually prevent harm to customers, that's a better solution anyway, since it encourages leaders of companies to behave in a manner that's the opposite of collusion.