Also, they may have signed up to find out more because they wanted to find out more right then. After they learned more, they no longer cared. Since they no longer cared and there was no presented way right then to indicate they don't care, they may look at further marketing emails as spam, especially when there is a lot of them and not a simple follow up 'we miss you' a couple of weeks later.
Just because you know you're not sending out spam, doesn't mean that what you are sending doesn't look like spam to the receiver.
That Google then turns around and treats my signal as an indication of the level of trustworthiness of the whole company, and then uses this level for other users is actually irritating to me. I'm just using the user interface in the most convenient manner to me. Yes, I should go and unsubscribe - but that's laborious, and clicking "Spam" isn't.
If I'm reading email on my phone I don't want to have to spend 30 seconds entering my email address or looking up the password for some account I haven't used in months. In the worst case I've lost my password or someone else has used my email address by mistake, and then I'm forced to find the password reset option, re-check my email, and change the password just to unsubscribe. By that point just clicking 'spam' and getting the email out of my inbox starts to sound like a good idea.
And that five seconds is a lowball estimate if the unsubscribe link is actually to a "manage your email notifications settings" page as is so common these days.
IMHO, unsubscribe links should take one click, and should be at the top of the message, clearly marked. Even if you accidentally click it, so what? Just have another link on the resulting unsubscribe page to immediately undo the unsubscribe. Also, having better support in mail clients for unsubscribe mechanisms (I know Mailman puts in a header with an unsubscribe link, clearly labelled), would be a good idea. I could even see spam handling systems getting a little more sophisticated (if they aren't already), and attempt to unsubscribe via methods in the email first, then if that same email address gets any more email from that source, autoblock it as spam.