The downside is you get embarrassed when someone makes a map. That's the NRA fear. (Edit: that's WHAT the NRA fear)
Similar downside for...driver's licenses, passports, political contributions, subscriptions to magazines or any other database where it is useful to have records of who does what.