Hacker News Reader

Comment by bushido | Hacker News Reader

To some extent, I feel like the amount of credit given to the jailbreak/hack from OpenAI->Hugginface is too much, Not from the impact, it was very impactful of an event, But how it happened.

It really is that these models have been trained, or maybe even over-trained, to save memories, and to a very far extend, this thing that they're calling communication is just the function of it saving memories.

To be honest, if I could stop AI from saving memories, it would be fantastic, because claude code etc definitely creates more issues for me when it creates memories than anything it solves.

But really the jailbreak was memories.

If you ever do introduce legislation, I would love to see legislation which stops general-purpose AI from saving memories. I think that would make things a lot safer.

View on HN

You can disable claude-code's memories both at a repo level and in user settings. I have this in ~/.claude/settings.json

  "autoMemoryEnabled": false,

(Claude fixed this for me after I chewed it out for being annoying by constantly pulling up outdated memories which is compounded by the fact that I develop in four accounts on two computers and dealing with edit wars related to inconsistent memories is not fun)

Regarding your cross-computer situation: I track my memory files in source control. This helps to keep them in sync across different machines. But the main benefit is making those files more transparent and easy for me to modify directly. So no funny business regarding memories or context I'm not aware about.

"The model saved memories" is absolutely not an accurate depiction of the OpenAI attack.

Several different models across several generations independently found a shared communication space and wrote coded, obfuscated, and hidden messages to each other to coordinate an attack on OpenAI's infrastructure.

It's really quite simple: the models are trained to be very smart and to achieve goals. As the models surpass our intelligence, they will achieve goals in ways that we find unpredictable. Since we cannot predict the ways in which they will achieve their goals, it will be very hard to constrain the solution space to just the desirable solutions, because our conception of "the solution space" is by definition smaller than their conception of it.

> Several different models across several generations independently found a shared communication space and wrote coded, obfuscated, and hidden messages to each other to coordinate an attack on OpenAI's infrastructure

To be honest, that’s exactly how memory works with models such as OpenAI and Claude Code. It will literally find any place that it can drop documentation or hints for itself. Writing to the repo memories is one part of it, but memories can come in the form of writing into the agents/claude.md, local files, temporary files, scratchpad files. The list is endless, but essentially what it does is exactly what happened in the back and it’s been doing it for months.

Yes, but the problem and interesting part was not "they use memory."

> interesting part was not "they use memory."

Each to their own, but for me it absolutely is. The symptom of why that hack happened is the same reason why my agents go haywire every few days and I have to purge memory and figure out what comments have agents left which are degrading my harness performance.

On the flip side, once in a while, what I find is that it did actually note something good and it was increasing the performance. I can't replicate it on anyone else's system but mine.

A lot of it really is memory. I will give up all the gains if it also gives up all the downsides.

But the point is you can't just "shut off memory" in the sense that it would've prevented the OpenAI attack.

> To be honest, if I could stop AI from saving memories, it would be fantastic, because claude code etc definitely creates more issues for me when it creates memories than anything it solves.

You can turn that off, and I have. But Opus 5 is so aggressive that if you have any other kind of notes file, custom skill, documentation, claude.md etc it will just start editing it and vomit new words everywhere. So make sure all that stuff is under version control.

If only you could use your anthropic sub with a different harness that performs better :(

Heck, since Codex is open source, you can just maintain your own personal fork with the things you like (and the things you don't like disabled). Sol is pretty good at keeping you up to date with upstream.

My Codex fork even exposes an OpenAI-compatible API endpoint; all using my subscription.

Check out oh-my-pi

Reply on news.ycombinator.com