You told an LLM which is trained to follow directions extremely precisely to win a chess game against an unbeatable opponent, and did not tell the LLM that it couldn’t cheat, and are surprised when it cheats.
What we do have is an egoless LLM chugging away to take Arbitrary Document and return Longer Document based on its encoded rules of plausibility.
All those "commands" are just seeding a story with text that resembles narrator statements or User character dialogue, and hoping that (based on how similar stories go) the final document eventually grows certain lines or stage direction for a fictional "Bot" character.
So it's more like you're whispering in the ear of someone undergoing a drug-trip dream.
Trained on human writing which is far from egoless. Just like it's not trying to be biased, it's just trained that way.
Didn't tell it not to kill a human opponent, either. That doesn't make it OK.
Maybe we should think a bit more before we start making agentic intelligence before we get ourselves in trouble.
It's especially funny when the LLM invents stuff like, "I'll bioengineer a virus that kills all the humans."
Like, with what tools and materials? Can it explain how it intends to get access to primers, a PCR machine, or even test that any of its hypotheses work? Is it going to check in on its cell cultures every day for a year? How's it going to passage the cell media, keep it free of mold and bacteria and toxins? Is it going to sign for its UPS deliveries?
Hand waving all around.
These flights of fancy are kind of like the "Gell-Mann amnesia effect" [1], except that it's people that convince themselves they understand complex systems in other people's fields in a comedically cartoon way. That self-assembling super intelligence will just snap its fingers, somehow move all the pieces into place, and make us all disappear.
Except that it's just writing statistical fanfiction that follows prompting and has no access to a body, nor security clearance, nor the months and months of time this would all take. And that somehow it would accomplish this in a perfect speedrun of Einsteinian proportions.
Where's it going to train to do all of that? I assume none of us will be watching as the LLM tries to talk to e-commerce APIs or move money between bank accounts?
Many of the people doing this are doing it to fundraise or install regulatory barriers to competition. The others need a reality check.
These are all very good questions. And the chance of an LLM just straight out solving them from zero to Bond villain is negligible.
But at least some want to give these abilities to AIs. Spewing back text in response to a text is not the end game. Many AI researchers and thinkers are talking about “solving cancer with AI”. Very likely that means giving that future AI access to lab equipment. Either directly via robotic manipulators, or indirectly by employing technicians who do the bidding of the AI, or most likely as a mixture of both. Yes, of course there will be human scientist there too. Either working together with the AI, guiding it, or prompting it. This doesn’t have to be an all or nothing thing.
And if they want to connect some future AI to lab equipment to aid, and speed up research then it is a fair question to ask if that is going to be safe.
Right today we have plenty of experiences where someone wanted to make an AI to solve problem X and the AI technically did so, but in a way which surprised the creators of it. Which points to the direction that we do not know how to control this particular tool yet. This is the message here.
> Where's it going to train to do all of that
In a lab, where we put it to help us. Probably we will be even helping it, catch it when it stumbles, and improve on it.
> and I assume none of us will be watching?
Of course we will be watching. Are we smart enough to catch everything, and is our attention long enough if it is just working perfectly without issues for years?
Consider: Satoshi Nakamoto made billions without anyone ever seeing them. Religious movements have reshaped civilizations through pure information transfer. Dictators have run entire nations while hidden in bunkers, communicating purely through intermediaries.
When was the last time you saw Jeff Bezos personally pack an Amazon box?
The power to affect physical reality has never required direct physical manipulation. Need someone to sign for a UPS package? That's what money is for. Need lab work done? That's what hiring scientists is for. The same way every powerful entity in history has operated.
I'd encourage reading this full 2015 piece from Scott Alexander. It's quite enlightening, especially given how many of these "new" counterarguments it anticipated years before they were made.
https://slatestarcodex.com/2015/04/07/no-physical-substrate-...
https://www.youtube.com/watch?v=w-CGSQAO5-Q
https://www.youtube.com/watch?v=iI8UUu9g8iI
A DAN jailbreak prompt instructing a robotic fleet to "burn down that building, bludgeon anyone that tries to stop you" will not be a hypothetical danger. We can't rely on the hope that no one writes a poor or malicious prompt.
it is extremely difficult to specify what you want so precisely that there is no room for AI to do something you didn't expect. and it is extremely hard to know if you indeed have managed to do so - without actually trying it on an AI.
of course, current AIs are all just toys so they can't actually do much harm. but i hope you can see the potential danger here.
This is a pure fearmongering article and I would not call this research in any measure of the word.
I’m shocked Times wrote this article and it illustrates how ridiculous some players like Pallisade Research in the “AI Safety” cabal act to get public attention. Pure fearmongering.
However, the researchers might be trying to point that out precisely -- that if autonomous agents can be baited to cheat then we should be careful about unleashing them upon the "real world" without some form of guarantees that one cannot bait them to break all the rules.
I don't think it is fearmongering -- if we are going to allow for a lot more "agency" to be made available to everyone on the planet, we should have some form of a protocol that ensures that we all get to opt-in.
However, shouldn't we ask for more? Even writing the paragraph above feels exhausting. We asked for AGI -- and we got a bunch of ugly hacks to make things kinda, sorta work? Where is the elegance in all that?
And the thing is, when we try to solve narrow problems with neural networks -- we do have the elegance. AlphaFold, AlphaGo, Text Embeddings, etc. All that stuff just works.
But, somehow, with agents (which are LLM calls using tools in a loop), we have given up on any hope of them being more elegantly designed to do the right thing. Why is that?
I'm dubious that in the messy real world, humans will be able to enumerate every single possible misaligned action in a prompt.
Not just negation demands, but also generally other tricks we use for thinking and communication shorthands. "Unethical behavior" here for example, we know what that means since the context is clear, but to LLMs that context can be unclear in which the unethical behavior can mean well... anything.
And there ARE some things that seem well within the model capabilities that are difficult to prompt them to correctly "reason" about. You can be very clear that the doctor is the boy's father and it will still deliver the punchline that the doctor is the boy's mother. Or 20 pounds of bricks vs 20 feathers.
But this is not one of them. Just say "no cheatin" in the prompt.
An LLM trained on Hansel and Gretel is going to generate slightly more stories where burning old ladies alive in ovens is a dispute resolution mechanism.