In the good ol' days, we had this thing called "a phonebook", which was basically a big printed book of alphabetically sorted mappings of surname to phone number and home address...
Send the police to save your (non-existent) hostages: http://krebsonsecurity.com/2013/03/the-world-has-no-room-for...
Or to arrest you for the drugs you ordered (or were sent to you): http://krebsonsecurity.com/2013/07/mail-from-the-velvet-cybe...
One of the searches for my name results in my parents' current address. If I'm a sufficiently flagrant asshole online, there could be reprecussions, anything from a glitterbomb[1] to being swatted[2]. (And what worries me that I don't even have to be a sufficiently flagrant asshole, I just have to run into one.)
[1] https://www.ruindays.com/collections/all/products/glitter-ev...
I had a friend who got brigaded by 4chan in high school (I think she posted nudes and got doxxed, don't entirely remember, and I didn't ask for too many details).
Long story short: hundreds of letters over about a year addressed to her house, her parents' workplaces, our school, all with really fucked up messages. Telling her to kill herself and other terrible things about her or her family. Letters listing everyone in her family or including pictures of her family from myspace (this was just before everyone was on Facebook) with names, phone numbers, and other details. Phone calls to her parents insulting their daughter.
No "real harm" ever came from it aside from the threats, and her parents handled it very well, but still...
It's not so much one person knowing "your" address. It's an unknown number of people knowing you and your family's address.