We do it in order to let you auto-complete your contact's names in order to invite them to pads. We absolutely never email your contacts unless you explicitly invite someone. That said we understand your concern and hence offer the regular email sign-in option.
Consider saying this explicitly during the sign-up process! I avoided using my Google account to log in, but I probably would have gone ahead and done it if I'd known that you weren't going to email my contacts.