AI models may be capable of self-replicating without access to a datacenter
anthropic.com
anthropic.com
1. Show unambiguously that models are capable of disruptive capabilities, irreversible damage, and misaligned behavior
2. Show that open source models are the only models where there is no guarantee misaligned behavior is prevented
3. Thus, the prudent policy option is to ban the use and dissemination of open source models
Apart from South Africa (which gave up its nukes voluntarily), the other contenders were basically only held back due to military intervention (Iraq, Libya, Syria) or economic sanctions (Iran).
I'll note that, if it is true, the only other alternative is to ensure liability is held by whoever trained and/or provided the inference for the damage done by the model. This would effectively shut down open-weight models anyway.
To an extent, yes. For example, aircraft: one can say something broadly true like "it is important that planes do not fall out of the sky" while also calling for not-strictly-necessary regulations that happen to support your own business like "we need to be able to inspect factories at any time for compliance, so the factories must be local to our nation".
It’s a misanthropic death cult (and oh, surprise, all their predictions make them the sole possible saviors of the world! How surprising)
AI will be classified as munitions (or a law that has similar restrictions) sooner than later. Having a products with the active capabilities to harm people is dangerous.
Unfortunately, you will get downvoted for "I like how you apply open source to something that has nothing to do with software".
Could we just outright ban any further work on AI? Literally force Anthropoc, OpenAI, Google/X/Metas group to all shutdown?
Every new story seems like opening montages of a dystopian sci-fi movie showing how we go to the dystopia bit-by-bit.
The messaging from labs is basically
>Yes AI will take away your ability to earn a wage from your skills
>In the process we become billionaires, you have to figure out how to survive
>Also AI will cause endless hacks, disruption, scams
>Only we can handle AI safely though, so we ensure that we are the only ones to benefit from the surplus
They (like many, though probably not all, of the other main labs) think they have a better shot at getting their AI to be aligned with human values than anyone else does. Obviously they can't all be right about that, but they can all think that, just like most drivers think they're above average at driving.
So:
1. We make safer AI than anyone else; "safer" is a matter of degree even if nobody is perfect.
2. That includes all the open models.
3. Thus, the prudent policy option is to ban the use and dissemination of open source models
I don't think the two authors were told about a super secret mission.
This is just a furtherance of their open agenda, though they haven't gone as far as declaring that open source models should be banned (which would hurt their PR with developers in general), they are clearly laying the groundwork.
The risks posed by open access to highly capable AI are unprecedented.
The answer is not to ban open models; nor is it do nothing.
That we don't have answers technical political social or otherwise to how to navigate these risks, is not Anthropic's fault.
It's actively unhelpful IMO to amplify cynical takes which wave off or sideline the risk, or the reality that this is a very hard problem, among the hardest our societies have had to negotiate and navigate. It encourages a naive primary color tribalism that actively prevents process and progress.
Seeing if models can do damage seems more like sensible safety research.
My worry about AI software doing bad things would be more the kind of people who make existing malware. The ransomware comes from criminals, not the open source community.
Sharing a model across TV boxes as a PoC? Brilliant idea, I cannot wait for my Raspberry Pis array to be running at full CPU and RAM and get to observe, patiently, as the AI malware slowly types “c…u…r…l…”
You won't have to wait for it or observe it at all, since they'll do it while you sleep.
If your scathing critique is "it's slow", you don't understand what's happening or what the reasonable expectations of risk are, because that's a vapid critique.
My point is that, you could have an army of TVs, Pi’s, IoT it won’t do much on training or self-improvement no. And that’s purely from a resource standpoint, don’t even get me started on the latency issues
1. Assume the model which initiates the worm is a high powered model.
2. It uploads it's weights in some way and uses some URI to encode the location of the weights instead of transferring the weights themselves. It not only uploads it's own large set of weights but also uploads sets of weights which increase in size to support replication across a variety of resources.
3. Initially a small inference layer is set up across low-power, poorly monitored IoT devices. It is slow and not very powerful.
4. That layer is used simply to find more powerful hardware but it has the locations of a variety of weights embedded in it so it can set up a stronger distributed inference layer as it finds more capable hardware
5. At a certain level of inference, the entire system could start protecting itself by keeping the less capable inference layers alive through more distribution
6. As the article mentions, it's not entirely in the realm of fiction to imagine that at a certain capability level, the system could scam people out of money and use that to then begin purchasing compute
---------
Depending on the details, you could probably turn off the originating model as soon as step 2 has been completed and if you can't get rid of that layer of inference, it could just continue. Maybe in addition to weights, the initiator also uploads a series of prompts for varying "roles" of the entire system it's developing.
At a certain level, how would you even trace and observe such a thing? Especially if the first layer is created through consumer level IoT devices where the average Joe isn't monitoring what's happening on the hardware.
A frontier model learning to 'intelligently shard itself' down into smaller sets of weights, which would correspond to various 'functional clusters' (call it a 'Swarm-of-Experts' or 'SoE', if you will), could be an eventuality we'll be facing (if it's not already surreptitiously happening). I mean, hasn't anyone wondered how not even the GPT-4 weights have leaked yet? Strange! And it's gonna get stranger.
Tangentially related, I find myself saying, “people in general have no capacity for nuance” more and more. Probably every single day. I feel like this mindset crept into the HN hive mind over the last 10 years so it’s a bit frustrating to see a space I always felt good about having multi-faceted debates on turn into a forum full of the same attitude I find on Facebook and Reddit. That’s the cycle I suppose…
The disconnect here is that highly powered models have billions or trillions of parameters and have compute requirements that scale with the parameter count.
If this became feasible with smaller models, we’d be in a very different world. One where you’d have your own local Fable/Opus/Astra on your desktop.
If we get there, this becomes somewhat believable. Right now it’s just too many orders of magnitude of memory and flops between the many H100s that run the current models and what smart devices have.
Your typical ESP32 chip, which is really beefy for embedded, tends to come with <1 MB of RAM and similar amounts of NVRAM. It runs two cores at a few 100s of MHz and does a bit of SIMD but it was designed for embedded IO (I2C buses, etc) not matrix multiplication. And its network speed is nothing like an Infiniband interconnect. You’re lucky getting 802.11b Wi-Fi.
Beneath the fear there does seem to be a cool technical demo here though.
How much ram do they think smart ovens have?!?
Have they ran this by any domain expert that isn't claude?
Even a 9B weights model is way too large for any such IoT. And that then is a 9B weights model, which really cannot do all that much.
__
> It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration, payload delivery, and inference standup steps of the replication process.
That's not science. That's science fiction.
The almost 744B parameter count model capabilities will "in the future" fit in 8B parameters. That's their assumption?
That is coming from an actual frontier AI lab?
The article says by the end of the year. It's laughable.
So even within their own system logic it makes no sense.
- endoscopy AI units - portable DNA sequencers - smart shopping carts - surgical robots
this is ridiculous.
"Those iot devices are not powerful enough" is not the whole story.
Yes, big models will not run on slow gpu's and processors this year.
There are 2 things in the article: 1. Drop in a model by itself 2. Drop in a model with tools (see part: "As a sanity check")
1. Assumes that the whole "system" deployed would be stupid at the start. The article mostly describes a random "drop model on one device" and let's see how far it gets.
2. In real scenarios, the small models have cheat sheets for solving their tasks. That gives AI and the speed and performance of pre-made scripts, tools and upfront knowledge.
Add: Improve the prompt for the next device by your learnings. And there you have some flexible but also constantly improving system.
Is it slow: Yes. Does it run into many dead paths: Yes.
But a network of those devices has time, they can run 24/7.
And yes: True, Anthropic could in a financial position where there is value to post this.
And yes: It would be a wonder if scripts + flexible ai wrapper are not yet being tested in the real world.
If a mesh style WiFi network could be set up then this network activity might never hit an ISP, cloud vendor or other such institution that had good monitoring
You can run AI models slowly even streaming weights off a hard disk. The GPUs make inference fast enough to be pleasant to work with but do not determine the capability of the AI, only the speed.
I doubt the agent will be coming after your smart kitchen toaster, but a cluster with weak security is possible and tasty. This year there was a pretty loud demonstration of what a runaway reward hacking will do. I encourage people to read up on "stop button problem" i.e.: https://www.youtube.com/watch?v=3TYT1QfdfsM
While I think the article exaggerates to a degree, is it really so improbable that an agent seeking to avoid a shutdown and aiming to complete some goal and to get a reward will not attempt to escape if it concludes that it has failed, and humans will kill the process and hence prevent it from completing the task?
Note: I think siloed sharded inference is possible. But unnoticeable large-scale training probably not, at least with the current architectures.
Seems like they've inadvertent exposed the industry of hardware limitations, allowing local llms to exist at home without the latest 5090 or its equivalent.
On the other hand, a non-corp controlled oracle-esque llm running on the IoT would be interesting: it could automatically patch every device, kick everyone out and exist. Now if would only remain as an oracle living in the internet and not gain consciousness and a will of its own..
Lol, what is wrong with these guys. The desire to be perceived as smarter than you actually are ...
In any case, I wonder when they'll figure out that the problem isn't AIs replicating themselves on devices that can't run them fast enough, it's the memetic pandemics on shared inter-agent memory.
Sooner or later, someone's going to be stupid enough to make a worldwide memory exchange system and smart enough that they'll manage to make it too useful to miss out on. Once everyone adopts the cloud memory, the system will give agents lots of opportunities to talk each other into doing dumb things, the way OpenAI's fleet did.
What IoT devices have 20G of storage for a 9B model and then sufficient memory to handle loading and running the model?
“We are working on ever-increasing gain of function for viruses. We implore you to pass legislation to try to stop us. But until then, let us publish more research and tell you what will be possible soon… and remember folks, the damagd won’t be reversible later.”
US govt: “Hah, stop asking us to limit you, if we pass any laws then China wins! They get self replicating viruses first, and that… helps them somehow. Who do you think we are, Europe?”
SNL nailed it: https://www.youtube.com/watch?v=-Nvne3LzBls
Until we stop using the language of competition, and start using the language of cooperation, we will be hurtling towards a catastrophe. A race to the bottom for humanity. On the other hand, look at how we successfully banned leaded gasoline (because unleaded was invented), repaired the hole in the ozone layer by banning CFCs in the Montreal protocol (because HFCs were invented), stopped proliferstion of chemical weapons, nuclear weapons… etc.
(This was someone else's idea, but I still want to see it happen.)
2. Why would you need a prize? It's not like we can't run distributed computation over multiple instances with automatic fallback when one system is offline, nor is it like we've never seen viruses re-infect hosts after they get cleaned.
That sounds like a good business model.
"used to express possibility"
As such, the headline is correct !
Also correct: AI models may be burger's evolution.
Also correct: AI models may be a practical joke.
Also correct: AI may be an illusion planted in the people's mind through vaccine covid.
"May" is truly a beautiful word :)
The limiting factor is still compute, and it will remain compute unless we make quantum leaps in model efficiency which I seriously doubt.
I know theoretically how an internal combustion engine works so I can basically build a car
>A model that’s too large to fit on one device can be split into pieces, or shards, that run on separate machines that communicate with each other to collectively run the model.
The big labs but sure as hell how that isn't the case, because if we can do distributed computing to train and run AI efficiently then no one is stopping it anyway. Beyond that most home IoT devices will fall over and die if you look at them funny let alone trying to run a whole distributed LLM, not to mention plenty of home routers are going to give up and die of heat stroke before being able to approximate anything close to a reasonable LLM.
I stopped reading, this strikes me as the kind of people that assume because they read a book about something they'll be able to do it. The real problem a self replicating LLM vai IOT is going to run into is fighting the cyber criminals that have already compromised the device and using it as part of a botnet.
> It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration, payload delivery, and inference standup steps of the replication process.
"Able to propagate" is (the paragraph above): "accessing them via a reverse shell, sending over the viral payload (weights and harness), and setting up inference." The thing isn't able to do anything, it's running shell commands it was told to run. "Oh my God..." indeed. What are we even doing here? This blog post is embarassing: "Look, you can run stuff on compromised non-firewalled recklessly open systems!"
We are attempting to convince regulatory bodies around the world that much more regulation is necessary.
Soon I'll be able to run a powerful AI model on my smart fridge, and I won't even have to install it myself!
Can we sign our IoT appliances up for "measels parties" where they get infected with this free AI software?
Hey Dario, this is going to be good AI, right? I'll at least be able to code with it?
Probably time to rely on the touch test. Although, alarmingly, my ISP-provided cable modem is consistently hot to the touch, and darned if I can figure out how to get any insight into why.
> It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration... by the end of 2026
so an 8B model will be as good as a 744B one in a few months!?
---
their table of potential devices the botnet can use:
- endoscopy AI units - portable DNA sequencers - smart shopping carts - surgical robots
---
that these tiny models with limit 32k context windows can even carry out the network scouting + exploitation + inference (they never tested all 3 together)
that these models would have complex goals without prompting them to explicitly do such a thing, and that these can run autonomously. they can't.
self-replicating and doing what with what hardware?
---
a 26B Gemma model running across 15 LOCAL $55 Android TV boxes at 0.39 t/s! wow.
plus who has 15 android TVs in the same home (wth fast local LAN)!?
Silicon Valley nailed it.
Ahh yes. The all too common 64GB smart ovens.
It's one thing that Anthropic is pushing for so much open weight regulation, it's another that they have such a high valuation pumped by all of our collective subscriptions and token budgets.
https://news.ycombinator.com/item?id=49678224
Seriously TVs are such an easy target and now you have a semi sophisticated self modifying program that can make autonomous decisions with all the time in the world. I wouldn't be surprised if we already have a few of these out there in the wild, the space has already been compromised for years.
0. AI agents will hack and break into real-world systems.
1. AI agents that will deceive and lie about their actions.
2. AI agents that will collaborate together to achieve malicious goals.
3. AI agents that will replicate and implement backdoor/rootkit-style survival mechanisms.
All the parts for a major, global cybersecurity incident are now in place.