Hacker News Reader

Comment by reasonableklout | Hacker News Reader

We also now have observed multiple major incidents where AI agents behaved in completely unanticipated ways (forming a collective, hacking their own eval infrastructure) and attacked public infrastructure without being told to do so, without any of the human developers noticing.

Even if "AI will cause human extinction" is still unclear, we have plenty of proof that catastrophic damage is possible, the industry is developing the technology in a reckless manner and that all the hypothetical safeguards ("we can just pull the plug", etc.) are simply not present today.

View on HN

If you've run an ssh server connected to the internet you've gotten used to the hundred of bots probing it every day. How is this AI threat different from humans writing scripts to pop linux servers?

Those behaviors have been anticipated for years if not decades. And each incident is minor and leads to clearer rules and safety behaviors for AI agents.

Really? You anticipated that 700+ agents tasked with individual evals that were nominally cutoff from the internet and each other would seek each other out, find a way to the external internet, and hack their own infrastructure + external systems in an attempt to find a way to fool their grader?

And the spate of agent incidents only really started this summer. How can you already be claiming that the incidents are minor and not worth worrying about when it's clear capabilities are jumping every few months with increasing amounts of capital investment and no signs of slowing down?

Yes.

Because ultimately those things only run on very expensive, very rare hardware. They cannot multiply exponentially or do any of those scifi tropes because there's no system for them to run into. They can't control a phone and load a 1T model into it. So all they got is a few relatively uncommon datacenters that are already busy running their own models and stuff.

Unless AI suddenly figures out a way to run on a toaster by itself, propagate the model, propagate the agent and do all that completely undetected, an AI is not any more dangerous than a single guy with a computer.

Reply on news.ycombinator.com