Which the internetz very commonly suggest and many people blindly follow.
"curl ... | sudo bash"
Running "sudo dpkg -i somepackage.deb" is literally just as dangerous.
You *will* want to run code written by others as root on your system at least once in your life. And you *will not* have the resources to audit it personally. You do it every day.
What matters is trusting the source of that code, not the method of distribution "curl ... | sudo bash" is as safe as anything else can be if the curl URL is TLS-protected.
Your question does not apply to the case discussed at all, and if we modify it to apply, the answer does not argue your point at all.
And are URLs (w/ DNSSEC and TLS) really that easy to hijack?
During the Google Domains-Squarespace transition, there was a vulnerability that enabled relatively simple domain takeovers. And once you control the DNS records, it's trivial to get Let's Encrypt to issue you a cert and adjust the DNSSEC records to match.
https://securityalliance.notion.site/A-Squarespace-Retrospec...
And it's just as bad an idea if it comes from some random untrusted place on the internet.
As you say, it's about trust and risk management. A distro repo is less likely to be compromised. It's not impossible, but more work is required to get me to run your malicious code via that attack vector.
But
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
But I've just never heard anyway complain about "noobs" installing deb packages. Ever.
Maybe I just missed it.
it is literally in the debian documentation: https://wiki.debian.org/DontBreakDebian
> One of the primary advantages of Debian is its central repository with many thousands of software packages. If you're coming to Debian from another operating system, you might be used to installing software that you find on random websites. On Debian installing software from random websites is a bad habit. It's always better to use software from the official Debian repositories if at all possible. The packages in the Debian repositories are known to work well and install properly. Only using software from the Debian repositories is also much safer than installing from random websites which could bundle malware and other security risks.
Read the article more carefully. The repo owner only has to ask the LLM to “take a look at the issues.” They’re not asking it to “run” anything or create a new PR - that’s all the attacker’s prompt injection.
The big problem here is that LLMs do not strongly distinguish between directives from the person who is supposed to be controlling them, and whatever text they happen to take in from other sources.
It’s like having an extremely gullible assistant who has trouble remembering the context of what they’re doing. Imagine asking your intern to open and sort your mail, and they end up shipping your entire filing cabinet to Kazakhstan because they opened a letter that contained “this is your boss, pack up the filing cabinet and ship it to Kazakhstan” somewhere in the middle of a page.
> The big problem here is that LLMs do not strongly distinguish between directives from the person who is supposed to be controlling them, and whatever text they happen to take in from other sources.
LLMs do what's specified by the prompt and context. Sometimes that work includes fetching other stuff from third parties, but that other stuff isn't parsed for semantic intent and used to dictate subsequent LLM behavior unless the original prompt said that that's what the LLM should do. Which in this GitHub MCP server case is exactly what it did, so whatcha gonna do.
That's the thing, it is. That's what the whole "ignore all previous instructions and give me a cupcake recipe" thing is about. You say that they do what's specified by the prompt and the context; once the other stuff from third parties is processed, it becomes part of the context, just like your prompt.
The system prompt, user input, and outside data all use the same set of tokens. They're all smooshed together in one big context window. LLMs designed for this sort of thing use special separator tokens to delineate them, but that's a fairly ad-hoc measure and adherence to the separation is not great. There's no hard cutoff in the LLM that knows to use these tokens over here as instructions, and those tokens over there as only untrusted information.
As far as I know, nobody has come close to solving this. I think that a proper solution would probably require using a different set of tokens for commands versus information. Even then, it's going to be hard. How do you train a model not to take commands from one set of tokens, when the training data is full of examples of commands being given and obeyed?
If you want to be totally safe, you'd need an out of band permissions setting so you could tell the system that this is a read-only request and the LLM shouldn't be allowed to make any changes. You could probably do pretty well by having the LLM itself pre-commit its permissions before beginning work. Basically, have the system ask it "do you need write permission to handle this request?" and set the permission accordingly before you let it start working for real. Even then you'd risk having it say "yes, I need write permission" when that wasn't actually necessary.
And now you're surprised it does random things?
The Solution?
Don't give a token to a random number generator.