Probably because this would require thorough code review every time you shipped an update to ascertain as opposed to just identifying whether you accessed a specific API or not.
What happened is that Google blocked our updates claiming we were doing something we were not doing, and even if we did access the entire phone book, the users agreed to it explicitly.
Another issue is that we would have been fine sharing a single contact with the PICK_CONTACT intent, however there's a bug open in Android since 2018 that causes developers to have to ask for the full READ_CONTACTS permission even just to pick a single contact[1].
So in our case Google did not respect the user choice, claimed we were using user data in a way we were not (without any proof whatsoever), still rejected the update even when we added what they asked us to add to our Privacy Policy and in the end they also don't even fix bugs in the Android codebase to allow developers to use more privacy-friendly APIs in their OS.
It appears to me that Google is asking companies to lie about their policies or otherwise make it their policy to retain contact data on their servers.
The only reason I can see for this is to make Google seem less bad when it comes to privacy compared to other companies because they can point to all these other companies you’re giving permission to store data with.