If you didn't click the button, then of course, you don't send the unsubscribe message.

If you did subscribe to a newsletter and no longer want to receive it (which is the majority of these cases), then the unsubscribe action is the logical thing to do.