Somewhat related is a concern I have in general as things get more "agentic" and related to the prompt injection concerns; without something like legally bullet-proof contracts, aren't we moving into territory of basically "employing" what could basically be "spies" at all levels from personal (i.e., AI company staff having access to your personal data/prompts/chats) to business/corporate espionage, to domestic and international state level actors who would also love to know what you are working on and what you are thinking/chatting about and maybe what your mental health challenges are that you are working through with an AI chat therapist.

I am not even certain if this issue can be solved since you are sending your prompts and activities to "someone else's computer", but I suspect if it is overlooked or hand-waved as insignificant, there will be a time when open, local models will become useful enough to allow most to jettison cloud AI providers.

I don't know about everyone else, but I am not at all confident in allowing access and sending my data to some AI company that may just do a rug pull once they have an actual virtual version of your mind in a kind of AI replication.

I'll just leave it at that point and not even go into the ramifications of that, e.g., "cybercrimes" being committed by "you", which is really the AI impersonator built based on everything you have told it and provide access to.