It is impossible to address all of the concerns, and it is impossible to predict what concerns may even exist. It will require mass deployment to fully understand the implications of it.
Allowing a stocastic dipshit to have unfettered access to your messages, photos location, passwords and payment info is not a good thing.
We cannot protect against prompt attacks now, so why roll out something that will have complete control over all your private stuff when we know its horrifically insecure?