If some of your contacts have this enabled then they can rebuild the network.
There was a case not so long ago where unrelated patients of a psychiatrist were suggested as friend to each other. The only possible link is that they all had the psychiatrist in their contact or vice-versa. The friending suggestion by itself was an outing of private medical information.