A significant difference is that it is not legal to own slaves or trade in slaves. That makes contracts for the sale of slaves inherently unenforceable.

It is not illegal to refrain from speaking. If you want to enter into a contract that requires you not to speak under certain circumstances there is probably nothing inherently unenforceable about it.

I'm pretty sure for example it would be legal for the Diogenes Club [1] to expel you if you violated their no talking rule, if someone would actually make a real Diogenes Club.

[1] https://en.wikipedia.org/wiki/Diogenes_Club