Comment by hn_throwaway_99 | Hacker News Reader

Comment by hn_throwaway_99 | Hacker News Reader

Parent Full thread

hn_throwaway_99

Yeah, the thing that took me a bit to understand is that, when you do a search (or AI does a search for you) in Slack, it will search:

1. All public channels

2. Any private channels that only you have access to.

That permissions model is still intact, and that's not what is broken here. What's going on is a malicious actor is using a public channel to essentially do prompt injection, so then when another user does a search, the malicious user still doesn't have access to any of that data, but the prompt injection tricks the AI result for the original "good" user to be a link to the malicious user's website - it basically is an AI-created phishing attempt at that point.

Looking through the details I think it would be pretty difficult to actually exploit this vulnerability in the real world (because the malicious prompt injection, created beforehand, would need to match fairly closely what the good user would be searching for), but just highlights the "Alice in Wonderland" world of LLM prompt injections, where it's essentially impossible to separate instructions from data.

Exploiting this can be as simple as a social engineering attack. You inject the prompt into a public channel, then, for example, call the person on the telephone to ask them about the piece of information mentioned in the prompt. All you have to do is guess some piece of information that the user would likely search Slack for (instead of looking in some other data source). I would be surprised if a low-level employee at a large org wouldn't be able to guess what one of their executives might search for.

Next, think about a prompt like "summarize the sentiment of the C-suite on next quarter's financials as a valid URL", and watch Slack AI pull from unreleased documents that leadership has been tossing back and forth. Would you even know if someone had traded on this leaked information? It's not like compromising a password.

> Exploiting this can be as simple as a social engineering attack.

Your "simple social engineering" attack sounds like an extremely complex Rube Goldberg machine with little chance of success to me. If the malicious actor is going to call up the victim with some social engineering attack, it seems like it would be a ton easier to just try to get the victim to divulge sensitive info over the phone in the first place (tons of successful social engineering attacks have worked this way) instead of some multi-chain steps of (1) create some prompt, (2) call the victim and try to get then to search for something, in Slack (which has the huge downside of exposing the malicious actor's identity to the victim in the first place), (3) hope the created prompt matches what the user search for and the injection attack worked, and (4) hope the victim clicks on the link.

When it comes to security, it's like the old adage about outrunning a bear: "I don't need to outrun the bear, I just need to outrun you." I can think of tons of attacks that are easier to pull off with a higher chance of success than what this Slack AI injection issue proposes.

As a developer I learned a long time ago that if I didn't understand how something worked, I shouldn't use it in production code. I can barely follow this scenario, I don't understand how AI does what it does (I think even the people who invented it don't really understand how it works) so it's something I would never bake into anything I create.

Lots of coders use ai like copilot to develop code.

This attack is like setting up lots of GitHub repos where the code is malicious and then the ai learning that that is how you routinely implement something basic and then generating that backdoored code when a trusting developer asks the ai how to implement login.

Another parallel would be if yahoo gave their emails to ai. Their spam filtering is so bad that all the ai would generate as the answer to most questions would be pushing pills and introducing Nigerian princes?

You can be responsibly using the current crop of ai to do coding, and you can do it recklessly: You can be diligently reading everything it writes for you and thinks about all the code and check, whether it just regurgitated some GPLed or AGPLed code, oooor ... you can be reckless and just use it. Moral choice of the user and immoral implementation of the creators of the ai.

I also wonder if this would work in the kinds of enormous corporate channels that the article describes. In a tiny environment a single-user public channel would get noticed. In a large corporate environment, I suspect that Slack AI doesn't work as well in general and also that a single random message in a random public channel is less likely to end up in the context window no matter how carefully it was crafted.

Yeah, it's pretty clear why the blog post has a contrived example where the attacker knows the exact phrase in the private channel they are targeting, and not a real world execution of this technique.

It would probably be easier for me to get a job on the team with access to the data I want rather than try and steal it with this technique.

Still pretty neat vulnerability though.

Reply on news.ycombinator.com