Most of the time where this shows up I never subscribed in the first place.

Gmail needs to handle that in a much better way to ensure they end up in spam for everyone else.

I'm not hitting unsubscribe if I didn't subscribe in the first place because it sends a signal to the sender that I don't want them to get.