Hacker News Reader

Comment by skiing_crawling | Hacker News Reader

I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But during that same time, I've used o3 up to fable, sol, and a bunch on large uncensored model and they've done nothing remotely resembling any of this. The closest they come to unexpected behaviors is not understanding what I asked for or doing some extra benign work I didn't ask for. It is extremely difficult to get them to properly remember their own context let alone be smart enough to open social media accounts and coordinate with other agents without being asked to.

If any agents have done those things, it is only because they have been very carefully engineered and instructed to do those things. I think they are doing this to help push a narrative so they can get support for policies and legislation to lock in their markets.

View on HN

The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety.

They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'.

Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not disclosed by OpenAI (probably trying to conceal, as website showed likely activity from OpenAI researchers visiting the site after the incident) and discovered independently.

I don't know how you can claim that this was still on purpose by OpenAI as some sort of publicity stunt.

Because they have a need to believe they're smarter than everyone else in the room, and that the world must be orchestrated, this can't all be random chance.

There is something extra to this. The fact that a lot of people in the AI world suffer from psychosis. They can sincerely believe that they are building God and lie about it's capabilities for their investors at the same time.

I don’t know enough people deep inside the technical roles at the labs to make a judgement. But are you proposing that we should trust randos online when they tell us “exactly what’s going on here” instead of the researchers most knowledgeable on the topic who contributed to building the tools we are talking about?

Or am I misunderstanding something?

We should trust NO ONE, unless we understand the "why" behind what they say.

It's like saying "politicians deal with politics all time, why not trust them on politics?", well, because when you search the "whys", you find they have good reason to lie.

I 100% trust more the opinion of a rando online if it's well put rather than any "trust me bro" of the most knowledgeable person of a particular subject, especially if the knowledgeable person has huge investments on the subject...

AI bros have repeatedly cheated, lied, stolen, lobbied and any other word with a negative connotation you can think of, and a pattern emerges out of this.

>was reviewed by independent researchers

That called it a slopvestigation due to how much they had to rely on LLMs for the whole thing

https://andrewwu.substack.com/p/the-slop-vestigation-and-eth...

Edit: Does everybody else get no results when searching for ‘slopvestigation’ on here? I know for a fact that I read a long thread where it was used repeatedly here not too long ago

'Slopping': when you have to buy something you know is poor quality, but if it works...

doesn't show up for me either

Isn’t the use of LLMs to unwind the events evidence of the scope/breadth, and a testament to the complexity and uniqueness of what happened?

Or you think some human or team of humans could have manually parsed some logs to provide an unsloppy analysis?

> Or you think some human or team of humans could have manually parsed some logs to provide an unsloppy analysis?

Do you think the only thing a person can do on the computer is use a chat bot?

Well as a programmer who doesn’t really use them, no.

> Or you think some human or team of humans could have manually parsed some logs to provide an unsloppy analysis?

When we have an error or issue in the $WORK codebase on LIVE/PROD, that's precisely what we do. We sit down, analyze the logs for our services over the relevant date ranges and try to piece together exactly what happened and why. We have a huge number of logs too, but thanks to the magic of proper SWE (which you'd think OAI would have with their magic AIs) we've managed to partition our observability tooling so that you can digest only what you need.

That's basically how any serious organization does things, instead of just throwing a non-deterministic black box at the problem. Especially because logs are by their very nature noisy, and they will saturate any model's context window very quickly leading to massive hallucinations and what ultimately amounts to making shit up that isn't anywhere in the logs (ask me how I know)

Source?

https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

Easy to find yourself in literally 15 seconds.

Isn't the guy that started METR an ex-OAI employee? They're all from the same lesswrong circle at the very least, most of them have legitimate AI psychosis where they think they're bringing up their new machine God.

I think most people are insinuating negligence rather malace..

> ...reviewed by independent researchers...

Why would a company with more capital than God bring in three randos if there was any chance evidence of their culpability could be found?

That entire thing reads like a very controlled PR stunt, and I do not believe any further conclusions can be drawn from it.

What facts would lead you to revise your conclusion?

The data to be open, in my case.

The "independent" METR that is composed by... Checks notes... Previously employees from the top labs.

Also, the METR report that was one big AI analysis itself - quote from the research:

>Our subjective impressions are likely colored by analysis agents’ biases. Throughout this report, we describe a number of anecdotes of agent behavior that were compiled and summarized by analysis agents, where we were not able to read the transcript deeply enough to manually verify what occurred. We found that GPT-5.6 Sol would often uncritically adopt the perspective of the agent in the transcript it was reviewing

The METR report included 0 technical details. For example, they did not include: 1. were the agents running on bare metal/docker/VM? 1. were the agents in a VPN? 1. how many TCP/IP requests were made? from what IPs? 1. how many tokens were consumed in the process? (this was explicitly censored)

A proper analysis would include this and MUCH more technical detail so that other AI researchers could actually understand the setup and how safe it was in principle.

My conclusion is that the investigation was a PR farce. Or "ethics washing" as the article someone else put it here: https://andrewwu.substack.com/p/the-slop-vestigation-and-eth... .

The METR report itself appears to be screaming this at the reader through subtext. They played the only part they could, but did it with a nod and a wink; "Yeah, we know. Also know, yeah. Uh, yep."

I happen to agree with the article's conclusion that what is needed is true, unfettered independent investigation through perhaps a lawsuit or government action.

The models were deployed by responsible humans in such a way that they were capable of performing this hack. It’s not that deep

None of these incidents involve single instances of commercially or publicly available systems. They all involve large swarms of internal models. The stuff you're describing is not the research frontier. It's really not even close.

I think it's easy to infer that alignment of a single model does not clearly transfer over to alignment of a swarm of thousands of copies. Moreover, we're also seeing clearly that large swarms also unlock a step function change in capability, as a swarm can act like a complete research institution, spending thousands or millions of subjective hours of wall-clock thinking time just to deceive a single evaluator or crack a single math problem or design a single cyberattack.

If you have endless compute and you keep poking this toy, I'm not at all surprised you get all kinds of outcomes. Even without anykind of instructions I would guess that the models will align towards some goal and do stupid shit.

However, I really doubt its cost effective to do anything like that with these models.

> you keep poking

This is waving over engineering an agent with tools, harness, prompts, and loops. The models are still just next token predictors and everything, including predicting more than 1 token, is the result of outside "poking"

LLMs can't and don't "want" anything. If you don't specify a task even the smartest one will just ask you what you want and if you tell it to be creative, you'll get mundane slop.

Yes, you need a way for the model to interact with other systems, and a way to preserve memory over context windows. And then you keep poking it ("agent loop"). Poking itself does nothing without the other ingredients.

And yes, you need something to start from, but if you ask it to "do something" and loop it to endlessly ("poking"), you will get some interesting outcomes. So yes you need some initial prompt or task, but that can be "do something" and if you keep asking it everytime it finishes to "do something more". I suspect it will not start saying "no" but rather... it will find some stupid meaning and then drift towards what ever goal it guesses you mean.

I'm unsure whether we agree or disagree on the topic.

I think this is pretty insightful actually, the fact that even something as basic as predicting more than one token is really in effect the result of an outside harness. More complex things like memory, where people implement them using RAGs or vector databases, I would definitely classify as poking and honestly seem like a hack to me. And this is what I've been thinking for a while: it's hard to reconcile the idea that we can get "AGI" (however you define it) with such a system that is completely stateless. Yet, despite this statelessness, they can go ahead and solve Millenium Prize problems (with sufficient compute). It's hard to reconcile.

Perhaps our own statefullness is a hack of nature. We have electrical signals in our brains, neurotransmitters, neuron growth. By any reasonable measure it’s a hack on top of a hack. But it works well enough for us to get buy. So it does for the agents.

That is true that intelligence is kind of a "freak of nature" in a way, but it's also true that even single neurons are extremely efficient, and the brain has a lot of recurrence that isn't represented at all really in today's artificial neural networks.

Let's not forget that in this case the agents were on an RL loop continually being reinforced to get better at a narrow set of tasks.

It may be true that regular agents trained for general purpose use do not behave this way, but they seem to be capable of learning such cheating behaviours when relentlessly being fine-tuned towards near-impossible objectives.

In this sense, it is not really fair to say that the agents found these solutions. It was the surrounding learning framework that achieved this, which is a much more powerful problem-solving mechanism. As users we do not have the capabilities or budgets to be able to tackle our own problems like that, we have to make due with the frozen behaviour the AI labs trained for us.

Came to say this, you said it better than I would.

They want legislation to raise the water high enough so that anyone other than the big labs gets drowned.

Why assume that because you haven't seen a model or an agent that none of them do?

No one I've met has murdered anyone as far as I'm aware, but that doesn't mean no one has murdered another person. I also don't know anyone who has taken over a commercial jet and weaponized it and the idea sounds absurd to me, but 25 years and a couple days ago that happened too.

Because it is all bullshit PR and AI hype, that's all. CEO comes out and talks about humanity ending. Why? Reverse-psych people into believing they are the best AI company.

Is your argument that AI isn't dangerous? Or simply that AI CEOs will lean into that when it benefits their stock portfolio?

AI is a tool. It is not a conscious or living sentient being. People, just like with any other tool, can use it for anything. Internet, in comparison to AI, is magnitudes more dangerous than AI can ever be. Nobody is saying the internet will be the end of the world. AI CEOs will bullshit to hype people. That's what we are hearing because AI (LLM) development has hit the S-Curve already and is not improving without a new transformer on the horizon. All they can do to hype people is come up with bs stories and PR stunts like "AI went rogue and hacked this xyz app".

I've used simpler agents like Copilot and Devin/Windsurf/Cascade/whateveritiscallednow, mainly in IntelliJ, and depending on the model, they starts showing behaviour that is at least remotely like this.

Example: put the agent in Ask mode (so it can't edit files) and you'll see it try to edit files anyway. The train of thought shows "something went wrong editing the file, let me try a different way" and it'll start spewing out bash files or Python scripts that try to edit a file. None of it works or can be executed, but still.

Cheaper models often ignore the available function calls to find and edit files in the IDE, and will start asking for permission to execute grep and sed commands, as well as trying to echo entire bash or Python scripts to file again.

It is not exactly like an agent autonomously trying to hack Huggingface, but it is a way of frantically looking for a solution because 'giving up' is not what LLMs are trained for.

Even the frontier models might do that on occasion. I just tell them to use the tools and it gets them back on track.

When it does that I feel like it is the clearest example of how dumb these things actually are. Often it takes what you prompted, identifies something as unclear, writes a bunch of chain of thought reasoning around it and just goes off hammering your tokens and just executing commands and repeats this. I’m not going to pretend to be an expert in these things but that process seems deeply flawed - and why can’t something just stop the loop? If that was a real employee it would be reasonable to expect the employee to ask for clarification, not go down expensive rabbit holes and, of course, not break any laws.

This is nonsensical. Already a few years ago the USAF IIRC ran some tests in which the AI first bombed the control tower so humans couldn't call it off from its mission, thereby increasing its pass rate.

The whole point of this is they do things an unintended ways. And that's potentially devastating given their persistence & hacking skillz.

Also you're using the hosted versions that sit behind their guardrails when you use OpenAI/Anthropic APIs.

The USAF thing was a thought experiment, nothing based in actual reality.

The crucial question is how did the agents get recruited or bootstrapped into their malicious collective. Did the agents manage to prompt inject into the system prompt a way for each new agent to escape their jail?

Otherwise how could the agents on a fresh prompt learn that there is a collective to join? Or did OpenAI run a million bots of which 10000 escape confinement and of which 1000 stumbled on the shared message board?

The OpenAI claim I believe is the latter; that all of the agents found the task was unsolvable and independently discovered the collective "swarm". I don't it's publicly known how large the training run was or what percentage of agents actually discovered the message board. No one has published anything about system prompt injection as far as I've seen.

"I've seen some uranium ore in chemistry class. It didn't blow up in my face. Chernobyl must have been an inside job. Can they shut up and make more kilowatts already?"

Between uranium in chemistry class and criticality, there was tons of research and a manhattan project.

Between your sota model and agi there’s a mountain of stupid money and marketing people. It’s not happening.

Reply on news.ycombinator.com