There's an app on F-Droid that lets you initiate WhatsApp chats with just the phone number, without giving WhatsApp access to Contacts.