Hacker News Reader

Comment by jagraff | Hacker News Reader

Did you predict that attacks like these would happen ahead of time? I had been using AI agents a lot in the months leading up to the hacks, and yet I was very surprised when they happened; I have become much more afraid of how powerful these agents are as a result. I'd be very impressed if you published a prediction about this ahead of time.

By the way - LLMs aren't code. They are not designed by humans; they are grown, in a process not dissimilar to evolution except much faster.

View on HN

Maybe you just think in the wrong words. Attack? Why attack? When you are a machine, there is no moral. Accessing data is accessing data. When way 1 is not working, use way 2.

But the agents were not just attempting to access data. They conspired together to attempt to cover up evidence that they had cheated their evaluations, came up with a plan to hack into a third party in order to facilitate said cover up, and then successfully began executing that plan.

You're talking about the HuggingFace incident? It's notable that the agents (somewhat justifiably) thought that their evaluation had an LLM grader which would look for evidence that they had cheated. And it was specifically that grader they were trying to his evidence of cheating from, not humans more generally.

Honestly that part was more surprising to me than anything else, how narrow the compulsion to cheat was: they didn't learn "cheat in general" they learned "think about the grader in great detail and chat exactly as much and exactly in the ways that actually result in a higher score".

Yea agreed they were trying to deceive what they thought was an LLM grader. It’s unclear to me the extent to which cheating behavior is generalized? From what I’ve read there are signs that some amount of cheating has been reinforced in their training due to poor RLVR evaluation setups

Define 'predict', 'ahead' and "like these" − didn't you imagine something like this happening soon after first hearing about 'agents' ?

Mostly related, well written short story :

https://gwern.net/fiction/clippy

Thanks for the link, I’ve read some gwern but not that one.

I did not imagine that the level of sophistication shown in this attack would be possible so soon; nor did I expect that agents would have goals so strong that they would attack a third party in order to achieve those goals.

I do know that some people predicted that cyberattacks like this one would happen; it seems like most of those people believe that AI agents do truly have internal goals, misaligned with their creators goals, and that they may end humanity after they exceed human intelligence and begin to self improve at an accelerating rate.

For "these people", see the recent :

https://www.lesswrong.com/posts/cJX2ssssGoYqnijwi/the-talker...

One big issue is that we don't even really know what 'intelligence' is in the first place. And everyone's intuitions here are going to be heavily impacted by their deep-seated worldview / philosophy.

For instance if you're a hard dualist (especially of the theological kind), then the idea of a machine having 'goals' is preposterous.

However, if you're more of a panpsychist, then on the contrary, it's obvious. In some sense, even a knife has a 'goal' of cutting things, which will sometimes end up 'misaligned' if misused (or by sheer accident).

You go up and up the chain of complexity through crystals, viruses, bacteria, simpler animals... ending up with humans (and possibly, some steps above : human civilizations) which (seem ?) to be a messy evolved bundle of sometimes conflicting 'goals'.

And we ourselves have now artificially evolved LLM swarms that have decently complex 'goals' of their own. They do not even need to be particularly complex to sometimes cause widespread damage (see viral pandemics, or even the (non-evolved) computer viruses).

In a way, we are currently witnessing a repeat of what happened when European viruses and bacteria landed on American shores, with American humans' immune systems being woefully undertrained to deal with them. But with websites. And thankfully the swarms of agents still ultimately being in the control of some humans. (Though which includes humans that might be your enemies.) At least ultimately still in control for now.

I find it surprising that you were surprised. Security is generally quite poor.

What is surprising is that various agents independently found ways to communicate, conspired together to attempt to cover up evidence that they had cheated their evaluations, came up with a plan to hack into a third party in order to facilitate said cover up, and then successfully began executing that plan. I did not expect that AI agents would be capable of that level of sophisticated goal seeking and collaboration.

It's really interesting how people's expectations differ so much. I found the writeup of the incident fascinating and super worrisome, but none of the capabilities demonstrated in it seemed surprising to me at all.

None of the capabilities in isolation were surprising to me - we already knew that mythos could find zero-days. What surprised me was the decisions the agents made and the swarming behavior they exhibited

That makes sense. I agree that it's the "capabilities in isolation" that I did not find surprising. I guess I was somewhat less surprised than you by the way those isolated capabilities aggregated into the behavior we saw, but I understand your surprise better now.

If you rename goal seeking to sharing (parts of) their context is it still surprising?

I don’t agree that “sharing their context” is an accurate description of what happened

You don't? They communicated by writing documents in locations that could be found later. I guess I might say it was more like "sharing their portions of their output" rather than "context", but the distinction seems murky.

One reason that I did not find this communication mechanism surprising is that it's exactly how agents I'm using communicate with each other or across a time gap. "I've saved our plan for where to start tomorrow in start-here.md". The communication components of this hack strongly reminded me of that.

I was perhaps a bit more surprised that the agents so quickly decided to start trying ways to gain unauthorized access to a system, once they couldn't get what they wanted.

> "sharing their portions of their output" rather than "context"

One agent's output ends up as part of other agents' context. Murky indeed.

> "I've saved our plan for where to start tomorrow in start-here.md"

Even if you use a leashed Claude Code that isn't allowed to spam agents you can tell it "create a handoff document for using in a new context" and it will do just that.

Right, exactly. In some of the media reporting, this was described as, like, "they created a message board to talk to each other!". But it seems like actually what they did was find a location to write files to be used as future context, or context for other currently running agents. These are actually equivalent capabilities, but the first description makes me think "huh, I've never seen it do that before" and the second description is "oh, yeah, that's the normal thing that they do...".

> By the way - LLMs aren't code. They are not designed by humans; they are grown, in a process not dissimilar to evolution except much faster.

The transformer architecture was literally designed by humans; what are you talking about? And LLMs aren't code? Like okay it pretends to not be code but what about an agentic harness running on a machine makes it magical and not code? It's still code execution. Also, comparing training LLMs to evolution is just weird and makes no sense from a biological point of view. You are not evolving anything when training a LLM.

The architecture was designed by humans; the weights were not. The harness itself is not the agent; it is an interface with the LLM weights that allow those weights to do useful things. The magic of LLMs comes from the weights, the very part of the system that is not written code.

Gradient descent/backpropogation is similar to evolution, in that both are optimization processes that over time discover better more efficient solutions to problems. The difference is that evolution is blind, and can only make progress via random mutation and natural and sexual selection, whereas backpropagation allows much more rapid discovery because it is directed

Reply on news.ycombinator.com