I personally think this could be worked out by comparing the sender of the email with the unsubscribe link with the recipient of the unsubscribe request and if they aren't related at a certain level, the unsubscribe request wouldn't go through.

Off course, there are probably problems with this, too.