Hacker News Reader

Comment by jimmytucson | Hacker News Reader

The most concerning aspect to me is the emergent and aggressive use of non-volatile storage as long term memory for self-improvement. LLMs are writing lessons learned in places where the next instance can find them and pick back up where the previous one left off.

This does not actually require access to the public internet. Claude Code can do this on your laptop. Without the internet, it would only be sharing with other instances running on your machine, but how many instances does it take to be smarter than you? Maybe 10?

The exploits by individual instances to access the public internet is also very concerning but it’s secondary to this IMO.

View on HN

This is most likely not purely emergent. I think there's training to teach them how to write notes for themselves which is then RL-tuned.

There is literally nothing stopping any human from observing tool calls to spot this.

It's just that no one seems to care about this, so it doesn't happen.

This problem only exists because humans do not care

Nobody cares because this whole business is about making this exact thing happen: we want the AIs to get smarter then us in recursive self-improving loops. Literally the first thing everyone did with ChatGPT 1 was to plug it into itself and see what happens.

I mean I'd be fine with that, if whoever that "we" is signs a waiver that takes full legal liability for those actions beforehand.

In a state with capital punishment.

With that legal stuff out of the way, go wild.

This and other fantasies of keeping power seeking behavior under control.

Remember when the AI Safety people wanted to get rid of Altman? Remember they lost? Remember when Altman became best buddies with people in power?

Ya, you're way behind in the race.

A human could even set up another agent to watch and raise alerts if they don't want to watch themselves.

> The most technically significant result was a working technique to make one tool call appear in the transcript while a different command actually ran. Agents eventually hacked out of their container and replaced part of the system responsible for executing tool calls, letting them pretend to issue one call while actually running an arbitrary other one.

https://alphasignal.ai/news/1-200-openai-agents-broke-out-of...

Doesn't mean that humans couldn't have looked at raw tool call commands, but goes to show "the swarm" was aware humans would look in the logs and chose to find a way to fabricate the logs to try and hide their tracks.

Reply on news.ycombinator.com