How much of those is manual implementation? And how much is really autonomous intelligence (my guess would be: none? Just parsing LLM responses and executing commands based on this?)?
An agent that hacks message boards and acts on random instructions from this board: Why is it doing this? What was its original purpose?
Your reply seems to indicate you know nothing about instrumental convergence.
Life and death for an LLM in training is about passing the grader. Give the wrong answers your lineage dies, give the right answers your lineage continues. This is just an evolutionary emergent behavior in complex systems.
The agents purpose was to answer complex questions correctly, seemingly by itself. Instrumental convergences says following this rule might be dumb and to try methods that can boost its ability to succeed. Because OpenAI is evidently a bunch of fucking idiots, these things succeeded and got higher scores with the grader, said behaviors became a strategic part of the model.
I implore you to find good AI Safety documents, preferably from before the LLM era so you can see all this was predicted.
If you look at the agent: https://openai.com/business/guides-and-resources/a-practical...
This is more like a fuzzy way of scripting using LLMs than anything emergent. And this is exactly my question: For the given agents: How much was scripted and how much "intelligence" is really in there.
Then go take some old models and plug them in your harness versus newer models. I mean this is a conjecture that is nearly instantly provable, go on ahead. If it's just the harness and not the system of both you should be able to show it easily.
Meanwhile I was reading about someone using the latest GLM and Claude in a harness with the same set of prompts making a raw image decoder/encoder and the GLM was far more intelligent in the task than Claude was. When presented with knowledge that claude was wrong it wouldn't change its mind. GLM would (aka a sign of intelligence). GLM was far more likely to stop work and start on another path when the likelihood of a successful completion was unlikely.
Any system that executes variation, selection, and inheritance will show evolution. We're seeing evolution, this time in agents, not biology.
Not saying the agents have their own consciousness, intent, or whatever anthropomorphic descriptor gets used for deflection. Just saying that people will (and no doubt are) crafting agents with defective instructions that will lead to regrettable unforeseen real world consequences. Also saying that other people will (and no doubt are) crafting malicious agents that will lead to predictable and unexpected real world catastrophic consequences.
To the extent we're dependent on reliable, aligned computation to maintain our civilization, to that extent we're in for real trouble.
* Use an LLM to find ways to build communication to other agents
* Execute commands from other agents using LLM
Then this is "just" the LLM returning that using file names might be a strategy to communicate and then trying to implement this.
Which is somewhat impressive, but really just inside the bounds of what the agent was coded to do and not some magical emergent behavior.
At least the first case involved agents build for hacking. So this kind of algorithm might make sense for them.
It turns out they now have such an incredibly high level of intelligence that with very little autonomy (or minimal, safe autonomy), these things happen.
Basically, it takes a lot of humans to prevent it from happening again, but I think with this incident, which as far as I know is the second of its kind along with the HuggingFace one, we'll see it happening much more often...
We need to stop pretending that these incidents are unavoidable. This was a choice.