This really is a false comparison. Accessing an address book for the app's use only and accessing the address book for public publishing are so far apart it's ridiculous. Security policies don't mean anything. If data is stored digitally there will always be a way to compromise its security no matter how much encryption, SSL, etc. you use. Security, especially on the web is a total misnomer. All you can really do is make it inconvenient to access.
Trusting a company isn't about their data privacy and security policies. It's about their brand to a large degree. Their track record and other peoples' experiences with the app. I'm in the camp that understands these policies and what they mean but thats not the reason I trust the company behind an app. I trust Google with my docs because they have a good track record. Facebook I trust as a necessary evil. Scratch that, I don't trust Facebook but I use it anyway because I'm banking on the odds. I think that's what it boils down to for most average folks. If the odds are that they'll have no trouble then that's a risk they're willing to take. If there's a breach of security and a bunch of people have their data exposed (but I'm not affected) well that makes me think twice not because of the breach itself but because of all the pile-on press coverage.
I'm one who firmly believes that people don't make their own minds up about this stuff. The average person's view of this whole app uploading address books thing is based purely in whichever side of the manufactured debate is the loudest and seemingly most expert-y. After watching how people use the web, their computers, and their smartphones for some time now I've become really cynical when it comes to this stuff. People don't seem to care until someone writes a blog post that tells them they should care.