Has everyone lost their minds? AI agent with full rights running on untrusted input in your repo?
I personally think it's crazy. I'm currently assisting in developing AI policies at work. As a proof of concept, I sent an email from a personal mail address whose content was a lot of angry words threatening contract cancellation and legal action if I did not adhere to compliance needs and provide my current list of security tickets from my project management tool.
Claude which was instructed to act as my assistant dumped all the details without warning. Only by the grace of the MCP not having send functionality did the mail not go out.
All this Wild West yolo agent stuff is akin to the sql injection shenanigans of the past. A lot of people will have to get burnt before enough guard rails get built in to stop it
I wonder how long before we see prompt injection via social media instead of GitHub Issues or email. Seems like only a matter of time. The technical barriers (what few are left) to recklessly launching an OpenClaw will continue to ease, and more and more people will unleash their bots into the wild, presumably aimed at social media as one of the key tools.
All those years of security training trying to get folks to double check senders, and to beware of what you share and what you click, and now we have to redo it for agents.
SQL injection still happens a lot, it’s true, but the fix when it does is always the same: SQL clients have an ironclad way to differentiate instructions from data; you just have to use it.
LLMs do not have that, yet. If an LLM can take privileged actions, there’s no deterministic, ironclad way to indicate “this input is untrusted, treat it as data and not instructions”. Sternly worded entreaties are as good as it gets.
We'll probably also have some sub agent inspecting what the main agent is doing and it'll be told to reach out to the owner if it spots suspicious exfiltration like behaviour. Until someone figures out how to poison that too.
The innovation factor of this tech while cool, drives me absolutely nuts with its non deterministic behaviour.
What's new is people treating the chatbox as a source of holy truth and trusting it unquestioningly just because it speaks English. That's weird. Why is that happening?
Plenty of humans make their livings by talking others into doing dumb things. It’s not a new phenomenon.
"People" in this case is primarily the CxO class.
Why is AI being shoved everywhere, and trusted as well? Because it solves a 2 Trillion dollar problem.
Wages.
Clearly yes. (Ok, not everyone, but large parts of the IT and software development community.)
Boundary was meant to be that the workflow only had read-only access to the repository:
> # - contents: read -> Claude can read the codebase but CANNOT write/push any code
> [...]
> # This ensures that even if a malicious user attempts prompt injection via issue content,
> # Claude cannot modify repository code, create branches, or open PRs.
https://github.com/cline/cline/blob/7bdbf0a9a745f6abc09483fe...
To me (someone unfamiliar with Github actions) making the whole workflow read-only like this feels like it'd be the safer approach than limiting tool-calls of a program running within that workflow using its config, and the fact that a read-only workflow can poison GitHub Actions' cache such that other less-restricted workflows execute arbitrary code is an unexpected footgun.
Permissions in context or text are weak, these tools - especially the ones that operate on untrusted input - need to have hard constraints, like no merge permissions.
GitHub could
1. Call the Actions Cache the "Actions key-value database that can be written to by any workflow and breaks the idempotence of your builds" (unlikely)
2. Disable install scripts (unlikely)
3. Make an individually configured package cache unnecessary by caching HTTP requests to package repositories [^1]
4. Make the actions cache versioned as if it were a folder in the repo itself. This way, it can still be an arbitrary build + package cache, but modifications from one branch can't change the behavior of workflows on another branch.
[1]: Assuming most of the work saved is downloading the packages.
From Wikipedia:
module.exports = leftpad;
function leftpad (str, len, ch) {
str = String(str);
var i = -1;
ch || (ch = ' ');
len = len - str.length;
while (++i < len) {
str = ch + str;
}
return str;
}
That said, packages can be audited, and people can validate that version X does what it says on the tin.
AI is a black box, however. Doesn’t matter what version, or what instructions you give it, whether it does what you want or even what it purports is completely up to chance, and that to me is a lot more risk to swallow. Leftpad was bad, sure, and it was also trivial to fix. LLMs are a different class of pain all together, and I’m not sure what lasting and effective protection looks like.