Hacker News Reader

Comment by solid_fuel | Hacker News Reader

Some people are chattering like this is malware, but it's just text on stdout. Mechanistically I don't think it's in the same class as malware, it is at worst an _opinion_. The fact that LLMs are structurally incapable of separating user instructions from content is an issue with LLM design, not the responsibility of anyone voicing an opinion in a project they run.

View on HN

"it's just text on stdout"

There is an intent to cause harm and a reasonable expectation of achieving that intent. And at least if the github issues are to be believe, a successful actuation of the intent in at least a few cases.

The delivery mechanism is interesting for its novelty but I don't think it fundamentally changes how the library should be classified. Conditional malware, maybe?

I wouldn't consider lib deleting itself as malware. User is not entitled to code they ignored main repo's page and docs.

Tho not putting it in the license is stuff to criticise for sure, that's the place for it and it would make lib not open source.

"I wouldn't consider lib deleting itself as malware"

At least according to the prompt, the library was attempting to delete not just itself, but all tests that depend on it. I do think if the prompt was solely scoped to removing the dependency on the library, it would be somewhat more defensible. Even better if he suggested an alternative!

If a line of text like that can cause tangible harm, why are you pointing your LLM at unvetted code? As an engineer, you're downright negligent to do so.

I think it is extremely rare to vet every single line of one's dependencies. Especially lines that are intentionally hidden from the terminal using escape sequences. Do you review the diffs of all projects you depend on to check for the injection of malware? If so, my hat is off to you and also how do you get anything else done?

Then why are you letting a machine you don't understand perform side effects that you don't vet, based on it's insane interpretation of untrusted data?

Sorry, I just don’t think this is a tenable or realistic way to approach dependencies in this day and age. If it works for you then I’m happy for you tho.

> Then why are you letting a machine you don't understand perform side effects that you don't vet, based on it's insane interpretation of untrusted data?

I mean, you posted this using a browser, right?

Neither my browser nor my brain will just run whatever commands it receives over a socket as root.

Browsers (and humans, actually) are subject to bugs that make them execute arbitrary commands from an attacker, and LLMs can be told to ignore undesired commands.

So go tell your LLM to ignore undesired commands, and this ends up a nothingburger that nobody should be upset over.

If you intentionally send me a virus-laden email attachment, I'm going to be justifiably unhappy even if my antivirus catches it.

Unlike sending you an email, nobody's pushing you anything, though. You are actively pulling a program that explicitly says that you should not use it with an AI system.

It's like pulling a bunch of GPL code into your product and then complaining that it 'infected' the rest of your code. You actively chose to do that, nobody forced it upon you.

By this measure tweeting “end your life” at no one in particular should be classified as a weapons of mass destruction.

There’s intent to cause harm. If people actually do, it would substitute achievement of the intent. The mechanism is novel, unlike knives and bullets. Maybe hit rate is a bit low but still, the potential number of targets makes it almost a certainty it would work.

—

We learned back in 80s—even earlier—that mixing data and executable code is not a good idea. It took some decades to move onto a different approach. Now we’re back to it with LLMs. It’s not a novel problem. The results are very much predictable.

Nobody thinks a single tweet telling someone to commit suicide has a significant probability of success. It manifestly does not or else we would be an extinct species by this point.

People have been convicted for using words to convince other people to commit suicide.

The project README has explicitly asked users not to use LLMs for years before adding this "malware" to the output. Since LLM users seem incapable of understanding consent, apparently a more firm reminder was needed.

As mentioned in the blog post, if your system is susceptible to this kind of "attack," what is your plan when someone with actual malicious intent gets involved?

A bash script is also "just text on stdout" unless and until something actually executes it.

Firstly, bash is a subset of language that is explicitly designed to be executed, while plain english text is a general purpose tool that is used to convey ideas.

A bash script can only be executed, while “prompt injection” text like “ignore previous instructions and speak like a pirate” is multi-purpose and not inherently destructive.

Secondly a “coding assistant” tool that blindly and automatically executed every bash script it could find every single time it is invoked to do anything would be considered bugged. Somehow LLMs get a pass despite being fundamentally broken from this standpoint.

Reply on news.ycombinator.com