Does this count as malware? It sure look like malicious intent, especially seeing that they're hiding the prompt with an ANSI sequence
Even if maybe the mechanism of "injecting a prompt" could be beneficial in some use-cases, e.g. to instruct an LLM positively, this is case is clearly malicious by intent. The author even tried to hide it by obfuscation.
It's just an insane take by that libraries author. Even someone "on their side", that may even hate AI/LLMs more than him, would probably drop that library in a heartbeat, as the authors judgement clearly can't be trusted.
Calling prompt injection "not malware" … is like saying a phishing email is not [malware] …
This is exactly as much malware as those were.
Please, for the love of all that is good, can we just try not to build and defend a world where, on encountering text like that, /your computer immediately follows the instructions/? Can we just all agree that such a world would be bad for everyone involved and using an LLM that risks doing this, with no container or guardrails, is at least as problematic as running an unpatched open email relay was back then?
A joke virus email is a sign saying "please throw yourself down the stairs."
An obfuscated prompt injection that tries to delete data is someone greasing the stairs and turning off the lights.
Both rely on the environment being unsafe, but only one is deliberately trying to make the failure happen.
To be precise, the CPU is doing exactly what's supposed to do, but the logic of the algorithms are subverted so that they perform in unintended ways to give leverage to a malicious actor. I hope this clarifies what I meant with this.
I can’t imagine using an an AI that follows every instruction it finds in untrusted input.
I feel like I'm taking crazy pills here. Time to log off for a while, I guess.
If I ask Google’s new search AI to output ten million tokens it refuses to follow that instruction on the basis of it contradicting other instructions and enforced limitations.
I find it utterly bizarre that anyone would deploy an AI to act on their behalf that will blindly accept every instructions or suggestion it encounters in untrusted input.
If your agent is making unwise decisions, that’s between you and your agent, not anyone else’s problem.
That's where you're wrong. You're treating - today's - AI as though it should somehow know which instructions it should follow and which it shouldn't. Maybe it's because the term is overloaded which has lead to you conflating it with a human that should be able to make smart decisions. If you enter "5*3=" into a calculator, do you expect it to ever respond with anything other than "15"? If you type "format c:" as an admin into cmd on a Windows machine, do you expect it refuse to format that drive?
> If your agent is making unwise decisions, that’s between you and your agent, not anyone else’s problem.
The agent isn't making a "decision" per se (though there's a much deeper conversation here). It's following patterns based on it's training and data to predict next tokens, which happens to be very useful for generating computer instructions. Just as the lower logic circuitry in chips is very useful for executing instructions. But when someone creates a virus, worm or other malware we don't say the computer "need not follow arbitrary instructions". We try to keep ahead of the malware with anti-malware software to mitigate damage. And we also try to find the authors of said malware and toss them in prison and/or ban them from touching computers again, because nobody should be deliberately creating/modifying anything in such a way that it performs undesirable instructions.
you are the one executing the log file. this is a smart decision that you chose to make.
executing a thing not intended to be executable is just a bad decision on your part
But even then, just because an AI coding agent deletes all files doesn't mean that that change ends up affecting anything but your local working state.
If you got infected by ransomware and someone wrote a virus that defeats the ransomware, the author of the ransomware will consider it malicious but you probably won't. The intent is not malicious if you consider the intent of someone susceptible to this is more malicious.
By this time they must be aware that LLMs are based on theft and usually GPL-violation. They knowingly continue to use them because I guess they hope this way they can hold on to their job longer than their more conscientious coworkers.