At the risk of sounding hyperbolic:

Imagine Iranian agents using these address books to track down naval officers in San Diego from across the border in Tijuana. Having a global address book lowers the barrier for hunting people down and hurting them. This is already happening to off-duty Russian officers mowing the lawn at home.

Being able to look up someone's address normally isn't a problem. Extremists and nut jobs being able to compile a list of people whose religion, political views, sexual preferences, medical conditions, and purchase habits makes them a target for violence is the problem. At that point being able to find them (using their street address or even real time geolocation data) becomes a pretty big part of that threat.