The service does not access your contacts unless you explicitly execute the import. You may choose not to import any contacts and add them by hand.

Also, I find it strange that people are more willing to trust the phone number than the list of contacts (which you can delete, anyway). Unless you store some credit card info in your contacts list, I guess :)