Whatsapp needs to know when you add a phone contact so it can lookup the corresponding whatsapp contact.

Even the 3 minute delay is annoying - it's common I meet someone, add them to my phone, and then want to send them a Whatsapp message and they don't appear in the list. I have to awkwardly hang around for a minute or so before being able to send them the message I wanted to send.

(for context, in most of Europe, Whatsapp is used pretty much like iMessage or SMS is used in the USA - wouldn't it be annoying not to be able to SMS a new phone number without waiting 3 minutes?)