are you sure the link works? I'm frustrated, but not really surprised, by the amount of email I get where the unsubscribe link returns a 404 or 500 error on submission. I'd say it's close to 20% of all unsubscribe links are non-functional.
Also, I've noticed a large number of "unsubscribe" links are actually "proof that someone is reading this mailbox" links. I've had email addresses I had to walk away from because I clicked too many unsubscribe links that resulted in lots of more "subscriptions"
My guess is that people have been burned by unsubscribe links so much that they mostly don't use them. I'd prefer to set up a filter on my side than have to trust that some spam sender is going to do the right thing.