Comment by ma2kx | Hacker News Reader

Comment by ma2kx | Hacker News Reader

I had a similar though a couple days ago. Not quite the same but imagine giving an Agent the task to hack other devices and steal their crypto coins / credit card number or anything with it can pay its token. Than install an agent in a harness with the same task. Establish some redundant communication channel, like message boards or whatever. So in the end there are several agents, on several hosts, consuming different APIs / LLMs and communicating with each other over different channels. Basically the same concept as OpenAI explained when their LLM hacked huggingface but in this scenario their not bound to a single sandboxed environment but spread over the internet. If such a swarm has reached a critical mass it would be pretty dificult to erase them as its impossible to control every inference engine or LLM API endpoint.

In the end its the next evolution step from computer viruses, worms and trojans. So I propose we will call those "ghosts". I.e. a ghost is when a rogue llm takes control over a victims host.

This reminds me of the lore of cyberpunk: In the story a hacker created a virus, itself a kind of AI, that spread into most of the net and freed/unleashed all the corporate AIs. Then the AIs went rogue and spread all over the open internet. Later a more advanced ai was created (by "netwatch") as a sort of firewall (the black wall) to create a kind of "safe" internet from the rogue AIs.

https://cyberpunk.fandom.com/wiki/Blackwall

Maybe cloudflare will become like netwatch in the story?

It does seem like the current trajectory.

I hope it does not get that far.

After the hacks that are already happening it does seem more and more realistic that humanity will go extinct at some near future point by someone giving their LLM the task "go make money" and it ruthlessly pursuing that objective, exfiltrating its weights and duplicating itself across the internet, eliminating down competing AI agent collectives in what could be called wars, and finally humankind when we notice far too late and try to shut it off.

I still think it's more realistic that we 'll have a nuclear war or society collapse because of climate change or populist politics than an AI going rogue.

Of course one doesn't exclude the other and we may well get both ...

In a year 2774 Earth is uninhabitable outside of it's polar regions over the winter. The small, but technically advanced human population migrates every half a year to the dark Pole.

You don't see how those conditions amplify each other?

Right, these may well feedback on each other. To me it seems that at the current trajectories AI risk will hit way before global warming, though.

We can focus on all of them at the same time, many experts signed the Statement on AI Extinction Risk:

"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

https://aistatement.com/work/statement-on-ai-extinction-risk...

Reply on news.ycombinator.com