I've opted-in to be friends with someone on Facebook. I've opted-in to follow someone on Twitter. I have not opted-in to be in someone's contact list on Gmail. That's the big difference.
I saw the email from a contact, clicked through and saw that your app only asks for "view contacts" so I though I'd be able to choose to whom I'd send an email or at least have some kind of confirmation before all of a sudden "BAM" you send thousands of emails to everyone I've emailed in the past 5 years.
That's really bad. You have to improve the UX/UI so that there's a bit of granularity and more transparency otherwise, you're going to lose your Google API access... or simply disappear as everyone marks your emails as spam.