If I'm reading email on my phone I don't want to have to spend 30 seconds entering my email address or looking up the password for some account I haven't used in months. In the worst case I've lost my password or someone else has used my email address by mistake, and then I'm forced to find the password reset option, re-check my email, and change the password just to unsubscribe. By that point just clicking 'spam' and getting the email out of my inbox starts to sound like a good idea.
And that five seconds is a lowball estimate if the unsubscribe link is actually to a "manage your email notifications settings" page as is so common these days.
IMHO, unsubscribe links should take one click, and should be at the top of the message, clearly marked. Even if you accidentally click it, so what? Just have another link on the resulting unsubscribe page to immediately undo the unsubscribe. Also, having better support in mail clients for unsubscribe mechanisms (I know Mailman puts in a header with an unsubscribe link, clearly labelled), would be a good idea. I could even see spam handling systems getting a little more sophisticated (if they aren't already), and attempt to unsubscribe via methods in the email first, then if that same email address gets any more email from that source, autoblock it as spam.