Suppose Alice and Bob are friends, and have each other in their contacts. Alice uses a simple, prepaid phone that only handles texts and calls. Bob has a smartphone.
Bob installs the Facebook app on his phone. The app wants permission to see his contacts, and he grants it. Maybe he's not paying attention, maybe there's a useful feature that needs his contacts, maybe the software asks the phone for permission, and the phone allows it without asking Bob. Whatever. Facebook now knows Alice's phone number.
When Alice gives Facebook her phone number to get a text (remember, that's all her phone can do), Facebook recognizes that it has seen this number before. Since Bob has this number, he is probably connected with Alice in some way.
Alice sees a recommendation for Bob when she logs on again.
Alice only gave permission for Facebook to text her. It didn't look through her phone at all.