And your friends are still using apps that harvest YOUR contact data. So your contact graph is still determinable. When I changed my phone I decided to not give out my new number. Now all of the spam calls come in on my Google Voice number that I share with everyone and I only share my iMessage with specific people I trust.
I'm going to guess that someone who doesn't use a phone's address book isn't going to use WhatsApp.
I use WhatsApp and deny it access to contacts, it just presents the phone number in the app instead. After a short time one's brain associates the 'last two' of the number with the person.
totally possible, make them write you first.
There's an app on F-Droid that lets you initiate WhatsApp chats with just the phone number, without giving WhatsApp access to Contacts.