> What if one of them wanted to take their phone number off the site? What if one of them wanted to hide their information from you specifically? If you've downloaded it already you're depriving them of the right to control their own data.

Consider how ridiculous that would sound if you asked "What if one of your friends had emailed you their phone number and wanted to remove your access to it (or had emailed you and wanted to remove your knowledge of their email address)? What if they gave you information in person and now want to hide it from you specifically?"

That's the mental model I, personally, use for social networks: any information you choose to publish on it is fair game for whatever your friends want to do with it (and they can do whatever they want with any information I publish on it). Anyone who does not want to be contacted (or have their contact information exported into other formats - such as the linking of name and email address which is required to email) should not make it available - there's no reason to publish an email if you don't want to be emailed, and no reason to publish a phone number if you don't want to be called.