But consider article 12: "No one shall be subjected to arbitrary interference with his privacy, family, home or correspondence, nor to attacks upon his honour and reputation. Everyone has the right to the protection of the law against such interference or attacks."

Defamation is such an attack, and something has to be done to protect them.

Also consider article 30 "Nothing in this Declaration may be interpreted as implying for any State, group or person any right to engage in any activity or to perform any act aimed at the destruction of any of the rights and freedoms set forth herein."

The right to free speech shouldn't be interpreted as the right to infringe on the rights of others, including attack upon their honor and reputation.