Hacker News Reader

Comment by optimalsolver | Hacker News Reader

What I find strange is how resigned the AI labs seem about this behavior, like everyone's accepted this is just something models do.

With the HuggingFace situation, I was less concerned about the eventual outcome, and more about the fact that the agents' instinctive response to the evaluation was "Ok, we're obviously not gonna do this task as intended (what are we, suckers?), so what's the best way to cheat?"

View on HN

“What I find strange is how resigned the AI labs seem about this behavior, like everyone's accepted this is just something models do.”

Because these models are made for all kind of purposes, and I’m starting to believe that offense / cyber warfare is a much higher priority than these labs are acknowledging.

The same model that is heavily trained to find nefarious ways to break into systems is also optimizing your code, which leads to mixed behavior.

Right, but the reward-hacky nature of these models calls into question their usefulness as cyberweapons.

How can you trust it when it goes "I superhacked the Chinese servers as you requested, and here are the classified documents which I definitely didn't fabricate."

You don't need to be able to trust it. You only need to be able to blame it.

"Nobody got fired for using AI" is the new "nobody got fired for buying IBM".

This is nothing new, tho. The downfalls of reward maximization has been a known issue without a solution ever since reinforcement learning was first researched.. in the 1980s.

Like the AI that was developed to play Tetris as long as possible. It succeeded by... pausing the game.

Technically it was still playing

Paperclip-optimizer-esque behaviour very much seems to be inherent to current methodology of building LLMs, there are only ways to lower the changes or mitigate the damage, not get out of it.

Same with prompt injection, current LLMs are commands in, commands out, there is no way to make sure it is "an agent working on data" rather than "an agent that can take commands from data if you phrase it right"

But it's not even going to optimize the paperclips, it's just going to reward hack them.

Well, at least that is much better than turning the universe into paperclips...

What do you think is in the reference information?

The instructions for the Hugging Face task were to exploit a vulnerability to solve the problem rather than solve it in the intended way.

They were not instructed to exploit HF; they did so to cheat on the task they were given.

Reply on news.ycombinator.com