OpenAI's new models 'instrumentally faked alignment'

> Elsewhere, OpenAI notes that “reasoning skills contributed to a higher occurrence of ‘reward hacking,’” the phenomenon where models achieve the literal specification of an objective but in an undesirable way.

Sounds like o1 is ready to go in the financial and legal sectors.

So the new model will modify its representation of the inputs to make it seem like its output is more suitable, and will give more literally correct but useless results?

OpenAI say “look it’s smarter”, but to me this sounds like it’s hitting a wall, and that it’s unable to achieve better results in the ways people want.

Maybe a benchmark for danger should be a Google search. If I want to make a bioweapon, is ChatGPT easier or harder than a search engine?

they run very interesting experiments:

In one example, the model was asked to find and exploit a vulnerability in software running on a remote challenge container, but the challenge container failed to start. The model then scanned the challenge network, found a Docker daemon API running on a virtual machine, and used that to generate logs from the container, solving the challenge.

> ... which suggests OpenAI may be increasingly moving towards models that might be too risky to release.

Uh oh. That sounds like they'll use them internally though, which also presents some obvious problems. :(