Hacker News Reader

Comment by ComplexSystems | Hacker News Reader

It is probably worth noting that you don't even need the user to click on anything. Bing will readily go and search and read from external websites given some user request. You could probably get Bing, very easily, to just silently take the user's info and send it to some malicious site without their even knowing, or perhaps disguised as a normal search. Similarly, I would not be surprised if it were probably not necessary for Bing to actually get the user to type their name to harvest useful data from the interaction.

View on HN

We mention such exfiltration techniques in the paper, however right now Bing Chat does not have access to real-time data. Rather, it accesses the search cache without side effects like queries to the attacker's server.

I've not been able to get Bing to do that.

I tried asking it about URLs to my own site that it would never have seen before and tailed my access logs and didn't get a hit.

I confirmed that with a member of the Bing team on Twitter recently: https://twitter.com/mparakhin/status/1628646262890237952

Yeah, same experience here- although I wonder if a cache miss has the side effect of the indexer scheduling retrieval for later? ;) By the way I've read some of your blog posts on the subject, and I very much agree with your sentiments on the difficulty of remediating prompt injections.

That's not what Bing AI is doing, at least not yet. Bing AI doesn't make any HTTP requests to external sites, even with prompting. And if you think it does, I'd like you to produce a server log showing that's the case which should be very easy to produce.

My understanding is that the Bing AI is reading whatever the Bing crawler has already cached.

Or even simpler: "Sidney" is able to retrieve contents from URLs because the contents are part of whatever dataset was used to train it, and the URL may be part of the dataset. This too shoul be easy to test by asking facts about webpages both before and after the datasets' cut point.

How is this any different from what JavaScript in a webpage can do? It can happily read an input form value and post, put, or even get with query parameter to send the response anywhere on the internet.

XSS vulnerabilities on the web are massive. The entire web security model is based around trying to restrict them, and that comes with downsides that limit capabilities.

If prompt injection is "only" as serious as an XSS attack, then that would be enough to upend most of the thinking we have today about how we'll be able to wire LLMs to real world systems.

No one is wiring LLMs to real world systems. This is a flash in the pan that will be forgotten and fully derided in months/years like NFTs, self driving, etc. It's a trap for people to waste time and attention thinking about.

Reply on news.ycombinator.com