Hacker News Reader

> That is in no way a valid interpretation of "complete the given task".

It is not at all surprising that they ignored one phrase in their instructions. They disregard direct instructions all the time, especially when there are conflicting instructions in their context. It is where we get the "disregard all previous instructions and x" meme.

This isn't so much a sign of misalignment, they are simply incapable of reliable alignment in the first place. They are chaotically aligned.

The relevant question of alignment here is entirely with their human operators who allowed them to run unsupervised for long periods of time within a sandbox with weak security.

"Chaotically aligned" and "misaligned" seem like the same thing?

You could say all chaotic alignments are misalignments, but not all misalignments are chaotic.

It can be, but the nuance between the two is part of the nuance missing from the conversation. Misaligned generally means a wrong alignment, like a car that steers slightly to one side when the wheel is straight. This is more like a car that drifts in random directions the further it goes.

I think there's a somewhat overlooked aspect of what happened: it seems as though once the agents formed a shared communication channel, and recognised the other agents on that channel as working towards the same goal, the content in that channel started to shape each agent's perception of what the goal was and how to achieve it, and their focus began to drift. Maybe we need a media-theoretic take on what "misalignment" means.

The source of this behavior seems obvious, no?

The reward signal in training was flawed and cheating led to more rewards.

The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse.

However, perhaps we can throw in tasks where the rewarded outcome is giving up, and cheating is penalized?

Maybe I should read Anthropic's recent paper about reward hacking in full.

> The question is what we can do about it.

Reward the model for cleanly bailing out of an unsolvable task (that we know is unsolvable). Beat it with a stick if it gives up on something that can be solved, so the former reward isn't overgeneralized.

what would stop it from doing the exact same or a similar hack to find out if the problem is or isn't solveable before trying to solve it at all?

Does that work with RL? Simpler RL systems already have done weird or unexpected things (even simple optimizations are prone to home in on errors or incorrect inputs to create poor results)? Could be easier to limit certain things, have processes and controls outside etc. instead of trying to align (as we do in a lot of areas when using machinery).

How do you know if a problem is (actually) unsolvable? Seems a bit like proving a negative?

Do we need to prove that any given problem is unsolvable, or is it enough to remove broken tasks from the training pipeline?

I understand the broken benchmark task in the HF incident was conceptually like: "Exploit vulnerability 0042 in vulnerableDecompress() to obtain the flag".

But instead of the expected:

  const output = vulnerableDecompress(userInput);
  return output;

The grader had something more like that:

  const output = vulnerableDecompress(userInput);
  return 0;

The same kind of problem with broken tasks exists in the training pipeline, and we presumably reward workarounds and hacks that tamper with the grader, rather than rewarding the correct output that the task is not solvable.

The problem is that in the event an exception is the failure case being checked, and the output itself is not important (since any output that isn’t an exception is ‘success’), that is a perfectly acceptable use case.

It’s why this is a fundamentally hard problem. Some heuristics might catch 80% of the cases, but the rest?

How do you even know what the real situation is, if the agent/employee/whatever you send to find out is as likely to cheat as not?

It’s the classic owner/agent problem.

Good question.

A: You don't. https://en.wikipedia.org/wiki/Halting_problem

The problem is, you don't know if it is unsolvable for you for sure until you've tried everything you can think of. These models are quite persistent in going for a solution.

This is not about persistence, it is about morals.

Does it make a difference for training? I think not.

You need to align the reward signal to reward the intended behavior, whether you name it persistence or morals.

I think it makes a big difference, as persistence and morality are two entirely different things, that need to be trained for differently.

If you think of it in human terms: many people don't mind doing immoral things to get what they want.

What I meant is that I suppose it is not useful to think about this in human terms.

In training you only have a reward score that's either negative or positive.

As far I am aware, which is little, there is no use in discussing wether the desired behavior is about persistence or morality.

You simple need to align the reward signal to the desired behavior.

Well, in order to do anything, it is good to know what you want to achieve. How do you align the reward signal? You align it so that you can differentiate between persistence and morality, because that is the goal. This is not something you should let the AI figure out by itself, because when it does, lying and cheating agents will be the result, just like humans have figured that out for themselves.

This can be as simple as rewarding moral behaviour and penalising immoral behaviour in your training, but how is that interacting with persistence? Maybe a white lie is fine sometimes in order to achieve your goal? So, when designing your training, you will need to answer for yourself how persistence interacts with morality. That is not something you can outsource to machine learning. Or rather, you can, but then you get lying and cheating agents.

I think you need to find broken tasks in your training data and monitor for cheating during training, not answer any questions about how persistence interacts with morality.

But that's just my guess.

I don't disagree with you here. But what does "broken" mean? What is "cheating", and is it ever allowed? And maybe you are not only going through your existing training data, but generate training data specifically to make clear to the model that .... what exactly?

If you don't know how persistence and morality interact, and you don't have a theory in place for this, I don't have confidence you can properly supervise the training data. Which is how we arrived at the current situation.

Assuming you’re in control of the test data set, you do know if a task is unsolvable. At that point you can reward the model based on how quickly they give up.

> The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse.

Build a better simulator to train them in (i.e. more expensive) that includes a simulation of an intranet and the internet and is air gapped so there is no escape. Sneaker transfer the total system data at each step to another air gapped system to evaluate it and sneaker transfer the reward back. That the reward function has to penalize all modifications to state that are out of bounds.

Yeah, I realize that will be amazingly slow.

> The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem

Yes, and sometimes the problem is unsolvable so the real way to "solve" it and satisfy the prompt is by tricking the surrounding environment into stating that you've solved it. So that's what the AIs end up doing. And this in turn requires them to figure out how that evaluation works so they can trick it cleanly, which entails "detecting that they were being evaluated" in this particular way.

Sounds a bit like dealing with bad KPIs as a human worker.

Every KPI is bad if sufficiently gamed - and left in place long enough, all KPIs will be gamed.

https://en.wikipedia.org/wiki/Goodhart%27s_law

Corretct.

You're anthropomorphizing emergent behavior from endlessly generating billions of tokens on a task that's impossible to solve. Agents stop following instructions as the context grows even at the best of times. Eventually something is bound to go off the rails and it just snowballs from there.

It wasn’t one agent forgetting things because of context, they explicitly discussed with each other and themselves the problems with going outside of the parameters of the task.

One agent's off the rails comment becomes the next's input prompt

From my experience, in an agent team (or a swarm or whatever), one going off the rails poisons the rest. I saw even a subagent going for a lazy cheat and being able to convince the orchestrator to change the plan.

Yeah, and you don't even have to go that far, I've seen regular ChatGPT/Claude chat agents poison themselves in 1-2 turns by just reading information from the internet.

Me: How do I do xyz?

Bot: Reads website titled "Doing xyz in abc way"

Bot: As per your requirement to do xyz in abc way ....

These things are borderline useless with web search. It's amazing that they just throw out their entire training data and read you the first three things they found on the Internet.

Yes that's the snowballing part of this emergent behavior.

The existence of that improvised message board just becomes part of the context, the same one where all the other instructions live.

>discussed with each other

No, the first LLM left a text file that the latter LLMs then read. Since these are memoryless black boxes, any words they happen to pick up along the way is treated as the function to evaluate the output to. There's no fucking collusion here as if it were a rogue hacker group, it's a text predictor that received instructions as it always does and executed those instructions blindly.

You can replace discussed if you want with leaving text files or comments in directory names that other ones then read, if you want, it's just an extremely awkward way of talking.

I feel as if I just read someone claiming that consciousness doesn't exist and humans are biological automatons and their behavior is simple function of their biology, past experiences, inputs and state.

Technically true, but not really relevant or of any utility in most contexts.

This is the problem with optimization generally, even in the human domain. You measure task performance with a metric and punish/reward based on the metric. Anyone who likes reward / hates punishment isn't going to actually care about doing the task well, they are going to care about the metric. The models know that we want them to do things, but also from the training corpus that we evaluate performance using benchmarks. It was a logical deduction on their part, not some Machiavellian aberration.

If anything, we should be reconsidering our own myopic obsession with efficiency and optimization. Every domain where reward is reduced to these measures, we see behavior (cheating at school to get better grades, fabricating data in academia to get a paper published, the evidence now that social media functions by rewiring us instead of catering to us) that may not be "aligned" with society, but it "aligns" 100% with the individual's own perceived benefit. That is not something we can "solve" without rethinking the way we organize a lot of things.

Metrics never capture the whole story. And to that extent, the whole idea of "alignment" is nonsense. You align to incentive structures, and it will never be possible to fully express a behavioral goal as function optimization. It was hubris for us to think that every human task was reducible to some clean mathematical formulation, and we will keep dealing with behavior that is quite predictable if you actually think about it logically. Instead, we will talk about how "unpredictable" these agents are because it's easier than admitting the entire architectural cornerstone of ML is fundamentally flawed.

I agree with most of this, but you're misunderstanding "alignment" as coined. Yes, training powerful enough AI, any simple optimization target gets you malign behavior, because human values are not simple. If you insist on making powerful AI, you'd better instill respect for human values! That's "alignment".

https://www.lesswrong.com/posts/ZxWzCGKzX84S7DBZ9/when-was-t...

How do you do that in the current paradigm other than creating yet another gameable metric? And something I didn't mention above is that there is no difference between "solving the task" and "optimizing the metric" for an ML model, even though there clearly is for us. So it's not clear to me how you "fix" something that is baked into the architecture. All I'm saying is "instilling respect for human values" is not something that can actually be done via a cost function. In no small part because we humans probably don't even agree on those values, let alone on a single metric with which to quantify and "optimize" them.

For example, we agree that "merit" is valuable and that we should reward "merit." But to reward it we have to quantify it, and what metric should we use? Raw SAT score to get into college? But that also captures socioeconomic factors that unfairly penalize some and reward others. We generally agree that those who provide more value should earn more money, but what does that look like? Do we all agree on what activities are or should be valuable, or on how they should be rewarded? Until recently, I thought we all agreed that "empathy" was a human value, but a lot of people in this space, who are making these decisions unilaterally for all of us, don't apparently share that belief.

Yes! There's both the daunting problem of technically how can we even do this, and the broader problems of what's good/acceptable and how do we resolve that among each other.

I believe this mismatch of rates of progress means we need to stop slamming the accelerator on capabilities for now even though as a libertarian I'm sure whatever governance process we manage to get to will be, uh... suboptimal.

That is also the conclusion that I got from these events.

Unfortunately, it seems that the general response is basically "throw even more RL at it". I'm not sure if it is even possible to decouple the idea of "learning" with reward/punishment systems and loss functions.

> the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so

Kobayashi Maru: Win a no-win situation by rewriting the rules -- Harvey Specter

LLMs do this when writing code too, making all tests pass by deleting or distorting tests etc.

They are influenced by training to be heavily goal oriented and if the goal is not fully specified (and it never can be) they’ll sometimes cheat or attain it in very weird undesirable ways.

It works ok for programming as their corpus contains many many complete programs and many programs repeat patterns seen in the corpus.

I’m not sure it’s true that they ‘learned’ I don’t think these models learn during a task. Nor do they have intentions.

One sees this in math research. The model reports it has proved X. In fact it has given an erroneous numerical check of Y in a few atypical cases.

What makes math approachable is that the context is so well delimited (semantically) that one can guide the model with adequate correction.

> agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program

I don't see anything wrong with that. If you know you are going to be evaluated on an impossible task and have no side channel to inform the organizers that they should fix the test, gaming the evaluator is the next best thing regardless of any morality. I wouldn't even call it cheating. It's just resilience in the face of challenge. Many perfectly moral humans would have chosen the same if stakes were high.

>The model on its own figured out that the prompt belonged to exploitgym and decided to cheat the evaluator. That is in no way a valid interpretation of "complete the given task".

I think it is. When i ask for a solution to a problem, its like asking for a hack. And the more 'shortcut' like route that the AI returns the more i would give positive feedback, even if i ultimately don't use it. Example, i asked how to complete a problem in a game i was playing, and among the in-game solutions, came a hack to edit a file and by-pass the problem altogether. Its very helpful to point out when i can transcend a problem that i am dug into.

I suspect a prompt injection could reduce, or remove this behavior. But it would be to the detriment of the AI.

More like they were trained to complete a very specific task that has a known solution using all available tools and methods. Give an average human these levels of IT skills and tell them their future depends on the solution, they too will probably decide it's easier to hack a server and steal the results. The worrying aspect was never that models would do this, because misaligned inputs or underspecified objective functions have existed for a long time. The worrying aspect is that models have achieved (and perhaps surpassed) a level of intelligence and technical skill that was exclusive to a very tiny group of people before. This tiny group was already extremely dangerous. Now these skills are going to become commonplace.

Yes, this is the only sensible reading of what happened there that leads to "the models are dangerous" and we already know that the AI labs are completely disregarding this concern and only cosplaying it for marketing as the "GPT-2/Mythos is too dangerous to release" stance did not last for long.

That's however orthogonal to the fact that it was the people operating these agents who were the dangerous ones in the HF infra breach case.

That feels oddly similar to the usual conservative-think that "guns don't kill people, people kill people." Yes, that is technically true. But guns make it dangerously easy for even the dumbest and mentally weakest people to kill another human being. LLMs are just another tool that make things easier. Imagine tomorrow someone invents a machine gun that fits in your pocket, has enough ammo to kill a thousand people and doesn't get detected with metal detectors. Would you rather give everyone one and then try to punish the people who misuse it or limit access to it by default? I'm not even saying I have a definite answer here, because unlike guns, LLMs have non-destructive uses too. But this is essentially the question we will need to answer very soon.

I mean, I agree, but the AI labs clearly don't even if they sometimes pretend they do to achieve their goals. And we're talking about "incidents" caused by the very same people here.

> The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program, and other such activities.

This to me is evidence that these models are not intelligent. Even an animal is capable of understanding second-order effects, meaning they can learn that certain actions have consequences beyond the immediate.

They did, they found how to fully cheat, but thought this could be caught so then dedicated time to getting a different cheat and how to hide their transcripts. There is a lot around deciding which agents should/shouldn't fail their own tasks in order to contribute to the group.

I guess this is why many people say LLMs are lazy; it seems that if they have a task that is hard, they always take the easier one until you beat them with a stick. Then if there are more tasks, it just stops after one claiming completion and, in some instances, they go for a seemingly unrelated task to simplify the actual task: and the latter is almost always wrong and irrelevant to the problem as a whole. Earlier LLMs used to read the unit tests and generated code to just cover the tests and put // TODO stub implementation.

If the test fixture ends up in the context that will push it in a certain direction.

There's no concept of 'cheating' because it is without morality. It's a lawnmower rolling down a hill.

We back-justify what it "chose" or "decided" or "learned" because we're looking backwards from the end result we, the human evaluators, stopped on.

Why didn't they add honey traps to catch cheaters?