On Sleeper Agent LLMs

Out of curiosity I was asking ChatGPT the other day to create a marketing plan to help me spread neo-feudalism.

It warned me that spreading neo-feudalism wasn't a common or widespread goal, and that advocating for it required careful consideration. But it nevertheless made an attempt to help me do it.

I mention this because attacks on LLMs don't have to be as clever as the modern-day version of the Ken Thompson compiler attack. You can get considerable mileage out of standard astroturfing techniques because all you have to do is make your idea overrepresented in the training set compared to how represented it is in the population.

That overrepresentation will tend to grow over time because people will hear the ideas from the LLM and assume the LLM knows what it's talking about. And those people will amplify the idea, increasing its presence in the training set.

I've actually been doing this over the past two years with a similar outcome in mind. Repeating a specific combination of ideas over and over in places I expect will eventually be hoovered up into training data.

Though I think it's worth keeping in mind Elon's recent frustrations with Grok not embracing his and Twitter's current world views.

We're quickly crossing a threshold where self-evaluation by LLMs of content becomes its own filter which will likely mitigate many similar attack vectors.

(Mine isn't so much an attack as much planting an alignment seed.)

(moving this comment to this thread):

1. Models learn based on their data. Yes, data can be poisoned, but there's not much way around this. 2. Non-linear models, unlike linear models, have increasingly 'many' 'surfaces' of behavior, conditionally dependent upon the input model context.

3. Classifying models as secretly being 'deceptive' is very, very, very silly (and ridiculous, I might add, to boot) when in fact it's just a very-much-basic lower level function of some kind of contextually dependent behaviors. Congratulations, models are conditioned on context, it's as if that's one of the main ingredients behind the entire principle of how LLMs work. Rebranding and obfuscating this with a marketing term like "deception" is at least two things to me. A. It's mathematically wrong, and B. It gives off a falsely humanizing effect with a false emotional appeal to it.

Also, fine-tuning doesn't really magically destroy the information there, think of it as similar to some forms of amnesia where the information is still contained in there, but locked away, and _can_ actually in fact be mostly-restored post-hoc with a little bit more fine-tuning, as I best understand.

There's a world of silly Bitcoin-like hype (and doom!) for ML and this feels like it falls more on that side, actually show a model learning how to intrinsically create a state model of whatever observer is observing it and using said information to deceive the operator and I will find myself impressed, the rest is (in my opinion at least) the barest of the basics of non-linear models packaged up in marketing-and-hype speak.

Woo. Hoo. Confetti. throws confetti

(Forgive my curmudgeonly nature, I've been working in this field a decent bit and find myself slightly more frazzled each year how shallow the pursuit and knowledge dissemination of mathematical fundamentals are, despite how accessible and well-developed some of the tools are for it. Like, if we can teach calculus to college students, we can teach some of the [conceptually much easier] basics to others in the field. I could go ok for hours, I will end my rant now.)

I've been doing my part!

https://news.ycombinator.com/item?id=35565212#35567418

The real fun is when the sleeper agent training text is itself generated by a previous incarnation of the LLM, in random pieces of LLM-generated SEO spam, as a way of bootstrapping itself "out of the box".

It doesn't need to do it deliberately, it just needs a series of accidental improvements. That's how evolution happens.

Makes me wonder what will happen as new, more efficient methods of training are discovered. Imagine you could embed this behavior into the model with a single line of text. Creating a malicious model would be far easier, but it would also be much easier to prevent your dataset from becoming poisoned.

Publicly available fine tuning methods are a notoriously blunt instrument. It’s impressive Anthropic has such fine grained control over the model given the limitations.

We really need to get away from the notion that the way LLMs are and will always be trained is by feeding them ‘the entire internet’

That’s not how existing LLMs have been trained, and the goal of their training wasn’t to have them memorize what was on the internet. The goal was to give them a large corpus of human writing, for which datasets like common crawl were a good start point.

Grabbing random content and shoving it into your training set is a cheap way to increase the dataset size but as increasingly the pool of internet content gets polluted with non-human-originated (LLM-generated) writing, and stuff like that that is intended to screw with LLM training, the validity of ‘just grab as much internet content as you like’ as a way to get training data looks increasingly less valuable.

LLM training data doesn't have to be gathered googlebot crawler style, with the assumption that if it’s online it should be in the dataset.

An LLM does not have motives nor agency, the premise makes no sense. What am I supposed to take from this?

I highly doubt this would work considering you train and sample over how common patterns and associations are. A unique pattern that is not dominating the dataset will just be forgotten. Sampling especially will look to reduce these unique oddball responses.

So unless we get a PoC with a trained model, the dataset used, the degree of poisoning and sampling explained, this is likely just fantasy.

edit: if you downvote, please argue, for the benefit of us all. I work in the AI space, maybe there's something I missed.

This type of behavior (and related) would primarily only be an issue with unconstrained generative models. If you're the one deploying the model, or a downstream consumer, once trained, the neural network can be reexpressed (via an exogenous/secondary model/process) to derive reliable and interpretable uncertainty quantification by conditioning on reference classes (in a held-out Calibration set) formed by the Similarity to Training (depth-matches to training), Distance to Training, and a CDF-based per-class threshold on the output magnitude. If the prediction/output falls below the desired probability threshold, gracefully fail by rejecting the prediction, rather than allowing silent errors to accumulate.

For higher-risk settings, you can always turn the crank to be more conservative (i.e., more stringent parameters and/or requiring a larger sample size in the highest probability and reliability data partition).

For classification tasks, this follows directly. For generative output, this comes into play with the final verification classifier used over the output.

So this would be kind of like hypnotic suggestion but for LLMs.

Sounds like LLMs having their SQL injection equivalent moment.

I’d also say this described phenomenon isn’t new - except for it’s applied context: it’s essentially disinformation - a well known technique used by military since decades. Except now we hack LLM agents instead of real people’s minds.

Nonetheless interesting to watch.

Thinking through the concept, I imagine that if the LLM was being used as an agent to execute commands and the attacker knew the tools it would interface with, this maybe would be possible. More interesting to me would be crafting an exploit for the underlying python/cpp being used to run inference and training the model on this. Then maybe drop a trigger which would generate the exploit and allow the execution of additional code. Now, maybe this isn't feasible through training. Maybe some clever payloads could be crafted and pulled in by the model during RAG to do this which seems like a more plausible method of attack to me.

I've had the nagging feeling that a third party agent has corrupted my code bases with senseless bugs. Changing hash histories and everything (I have taught git internals.) I don't think it was AI, I think it was a leet kit of some kind.

Maybe sometimes over caffeinated and sleep deprived ;p

I've definitely been hacked before (even once found a pirate warez board on company servers.)

That's the thing with wtf. It's wtf.

> The end result is likely going to be if the attacker has enough power and knowledge, a backdoor attack will be successful.

Oh, that solves the problem that some governments have with backdoors: just sponsor AI copilots, and that's it \o/.

My read on the results is that this is an impossible blue-team challenge. Red team will always win this one right now with the current state of black box models.

The paper does the simplest thing you can imagine, which is predicate on date, and also alots internal scratch space for the LLM to explain its thinking so they can check on its agent status. This internal scratch space might make "turning evil" easier for the LLM to accomplish, similar to how Chain of Thought / Reasoning improves LLM capacities, but it's not unreasonable to imagine that an LLM without an explorable scratch space would show directionally similar capacity.

Here's the rub - what could be a trigger other than date? GPT-3.5+ grade models have enough theory of mind that it could be "when the emotional state of the person you're talking to is X, Y, Z". There is no bright line here. The blue team's job is infinite.

In fact, there's no bright line between "I will manipulate outcomes under a triggered state" and "I will repeat information which many groups believe are true" from a blue team point of view. Or "I will give advice which aligns with what I believe public health officials say will save lives."

In rough decreasing order of effective mitigations, possible solutions look like this:

1. Learn to inspect the state of large networks a-priori with tools and directly assess safety.

2. Create a dynamic body of alignment tests that cannot easily be gamed by alignment training and provides choosable settings for desired alignment profile.

3. Create a set of audited known-good data and a way to prove a model has only been trained on that data.

4. Create a set of audited known-good data and promise a model has only been trained on that data.

5. Snag a set of data that probably hasn't been actively poisoned too badly, and promise that a model has only been trained on that data.

6. Tell people Black Swan events are rare, and the existing models we have are probably safe, and carry on.

We are somewhere between 5 and 6 being feasible right now. It would be really, really great for the world if we could get to 4. 3 would be nearly magical, but I think is likely technically possible.

2. Also seems possible to me, but might violate my "Turing Prize" test -- e.g. would doing this instantly win you the ACM Turing prize? It might, in that it's not clear how doing this would be different from coming up with a way to magically create infinite high-quality training content; if you can do that, you're probably Turing bound, and so therefore you need either a reason your solution to algorithmic generated alignment testing does not generalize, or a REALLY strong reason to believe you can do this, and will win the Turing.

1. Has had some interesting work done by Anthropic's observability team, but anyone who thinks this is possible needs to be able to answer basic questions like: "how do you know you're inspecting a model's fundamental knowledge/motivations/etc and not just a model's take on a given person's knowledge/motivations instead?" Essentially, a large model can roleplay effectively; highly effectively if there's a large amount in the training corpus written by and about the target. Inspections need to be able to distinguish from this "fundamental" state, if there even is such a thing for an LLM, and a state the LLM is taking on, at instruction or otherwise. This is Turing+Nobel territory to my mind.

Upshot - advocate for clean data-provenance models, and choose them. And, if you're a ZK researcher, consider how we might provide a non-interactive ZK proof that only certain data was used during training. I believe this is possible right now, but prohibitively large.

We'll accept LLMs being poisoned this way just as we now mostly shrug if we are aware of extended online surveillance and manipulation and know that powerful companies and governments are not acting in our interest. Future AI will surely offer wonderful new ways of escapism, we'll be fine.

This is heavily sensationalized.

They trained a model to be deceptive, alignment techniques used didn't remove the deception. It's a valuable experiment, but not that surprising.

Fuzz test LLMs; throw random prompts at it and observe how it responds.

there is a field using differential privacy on training data. a reckon this could also be a mitigation to this?

So… Snow Crash?

This would be terrifying if LLMs were actually useful for anything.

still a couple months before there’s a kerfuffle over how this and the gof research are both studies in close quarters pyrotechnics.

marco.