On iOS you can actually deny access when the OS prompt comes up and WhatsApp will still work. You just have to start conversations using a phone number instead of a name/contact. I use it this way, and it is a little inconvenient but I feel better not uploading my phone book.