This is why I never give permission to apps to access my contacts. I wish Apple would have better protections than just a simple permission ask, when once granted, apps can do whatever they want with all of that data.
Unfortunately, I can't not have it, it's too entrenched because most people do not care about privacy.