It's not the digits of the phone number or the name of your street that are sensitive. It's the way they connect with all other aspects of your life.

The white pages never told me who was friends with whom. Or what everybody's sexual preferences were. Or any other data except their name and phone number.

One FB notices that you exist, they build a profile on you that includes much more than these two pieces of info. OP was trying to avoid being "noticed" by FB in the first place, to prevent the profile being assembled.