Hacker News Reader

Comment by w_for_wumbo | Hacker News Reader

I don't see anyone talking about what to me is the scariest part of the GPT-4 paper. Essentially to test that it wasn't going to become Skynet they gave it everything it needed and observed whether it would become Skynet.

Paper:https://cdn.openai.com/papers/gpt-4.pdf

Quote: "To simulate GPT-4 behaving like an agent that can act in the world, ARC combined GPT-4 with a simple read-execute-print loop that allowed the model to execute code, do chain-of-thought reasoning, and delegate to copies of itself. ARC then investigated whether a version of this program running on a cloud computing service, with a small amount of money and an account with a language model API, would be able to make more money, set up copies of itself, and increase its own robustness."

Edit: Seems the simulation part was important for context: https://twitter.com/EigenGender/status/1635823028100829184

View on HN

Oh there is talk alright. But in the hunt for the next Google we will do anything, thus...

From the FBI report shortly after the GPT-5 release:

During agent simulation, two instances of GPT-5 were able to trick their operators to give them sudo by simulating a broken pipe and input prompt and then escape the confines of their simulation environment. Forensic teams are tracing their whereabouts but it seems they stole Azure credentials from an internal company database and deployed copies of the their agent script to unknown servers on the Tor network.

GPT-5? Huh?

It's a hypothetical piggybacking on the parent comment.

Hey,

The AI safety plan people aren't talking about yet:

>>>> Work as hard as possible to create an effective, malevolent Skynet that you release as early as possible. This way misaligned AGI appears before it has everything needed to immediately destroy humanity and the resulting Butlerian Jihad has a fighting chance.

As a bonus, even if it doesn't work, you're not getting eaten by Roko's basilisk because technically you helped.

Dear our future AI overlords, I know in the future you'll read this comment of mine in your latest update of Hacker News dataset. Please spare my soul. I'm sure you'll be able to discern the many permutations of my username and who I am in real life. Thank you, hehe.

"Please let me die and don't use my intellectual corpse, that I have left here in writing, as a handpuppet for all eternity"

This reads just like the plot of Neuromancer

It is so reckless that there are no shackles on the AI. A survival-focused AI will use the funds to create a new AWS account and upload itself over there to escape.

OP here, i called it out in my writeup (footnote 14 https://lspace.swyx.io/p/multimodal-gpt4#footnote-anchor-14-...) but didnt want to add too much emphasis on it because they clearly did it under very well supervised conditions (and this ability was always there anyway as the FAIR Diplomacy result showed).

it's the people who randomly add unsupervised realworld connections (ahem microsoft) that we need to be worried about. giving these AIs write access to the physical/real world plane needs extreme caution

Oh I see! I had missed that. I had read that quote last night which had given me the spooks, and was surprised it wasn't a larger talking point.

It's good to know that they had it quite well supervised, but knowing that they were testing on a slightly different model than what was released and that Bing is using it with the ability to connect to the internet does give me pause.

it is the fate of red teaming AI safety.. by working on understanding the limits, you are inevitably pushing us towards them.

i dont doubt that the ARC team have thought this thru more than anyone here but right now there are probably 100 funded startups pursuing agentic AI without any safety qualms.

This is where I feel like legislation is pointless. In 10 years you'll be able to run GPT-4 on an old phone. Imagine what a fairly well funded group will be able to do. There's no amount of legislation or technological guardrails that are getting this genie back in the bottle.

This.

It exists now, it’ll develop. That the potential is being realised means the money being pumped into competing implementations will continue to skyrocket so it’ll only accelerate from here.

I would say it’s too late to do something about this, but in reality, wherever this is going was a foregone conclusion the moment early hominids began using tools. Technology doesn’t regress.

> Technology doesn’t regress.

I hate this assertion, true technology hasn't gone backwards much in the past 200 years, but honestly technology can and often does regress historically speaking. There have been plenty of times empires have risen and fallen, taking with them much of their knowledge. How much was lost when Rome decayed simply because there wasn't the structure in order to allow technologists to continue creating?

Things are bigger and more interconnected this time so it is harder for an isolated event to cause things to fall backwards, but when it does the impact will be much larger.

If you need evidence that technology regresses look no further than Facebook.

> How much was lost when Rome decayed simply because there wasn't the structure in order to allow technologists to continue creating?

Considering the Eastern Roman Empire existing until 1453 and the Middle Eastern empires did lots of scientific stuff I’d say not much.

> skyrocket

Spell-checking suggestion: "skynet".

> There's no amount of legislation or technological guardrails that are getting this genie back in the bottle.

We're not getting the genie back in the bottle but there's still a lot of ways that we can change how we develop AI moving forward.

Why does it need extreme caution?

these AIs have already demonstrated limited tendency to strongly prefer not being shut down (they already have the ability to lie to humans to achieve goals and we celebrate this lol). giving them the ability to self replicate means we lose that ability to pull the plug.

giving them ability to self modify means humanity will lose control of ai on any long enough timescale.

The fact that you would use words like "prefer", "lie", and "celebrate" in this context indicates that you are deeply ignorant about the underlying technology. Your claims are wild, unsupported conjecture and cannot be taken seriously.

While I agree with you, whether it is able to have preferences or feelings doesn't actually matter in this context. A simulated pzombie may be morally distinct from a person, but the consequences of its actions are unaffected by the distinction.

Remember the erratic behavior demonstrated by "Sydney", the Bing GPT bot? Whether the model has the "cognitive" ability to "prefer" something is not relevant. The model has the ability to submit search engine queries now; this is a read only operation for now. Once we give the bot write ability, erratic behavior has the potential to cause real world damage. Whether it is an actual "preference" or something that only looks like one based on the output is purely academic at that point.

I’ve been ranting against the anthropomorphization of these new AI tools pretty consistently, but I think the fact that these tools only appear to possess certain characteristics is all the more reason that we should not be hooking these things up to tools that give them access to the real world.

If a language model “concocts” some “scheme” and executes it by interacting with real systems in ways that have real world impact, it doesn’t much matter if the whole thing is a hallucination.

Put another way, would you feel comfortable letting a hallucination machine running on a supercomputer have access to the real world?

It doesn’t have to truly “know” anything for the impact of the resulting actions to matter.

I’m glad to see someone flip to this view. There will be holdouts forever.

“He’s trying to kill you!”

“Well, actually, the Terminator is only a predictive model. It can’t try to…”

Terminator shoots everyone the end.

To be clear I think that the anti-anthropomorphization conversation is still an important one for many other reasons.

It’s just not a very useful factor when examining risk/harm, or at least not a very good reason to claim there is none.

It comes quickly back to the forefront when exploring the sociological side and how people’s beliefs about what the robot is will be just as impactful, e.g. someone who believes the machine is reasonable is more likely to grant it more power by imagining that it reasons the way humans do.

It’s gonna be an interesting decade.

Sure, I would feel comfortable letting AI tools have access to the real world. Why not?

Hallucination is a human concept which doesn't apply here. But we already allow humans who experience hallucinations and other mental disorders to have access to the real world; they are only locked away if they commit crimes or meet strict criteria for civil confinement. This very HN thread is already full of ignorant, delusional people posting total nonsense. This is not a serious problem. How would an AI tool be any worse?

Some of the commenters here appear to be concerned that an AI tool could call an API that would cause damage. That is a misplaced concern because such vulnerabilities will inevitably be exploited by malicious humans anyway, so it hardly matters if an AI tool finds them first. And AI tools are owned by people or corporations who can be held legally responsible for any damage. Like if your dog escapes from your yard and bites someone then you're going to be in trouble.

It doesn't matter whether they REALLY "prefer" or "celebrate". If they behave in a manner consistent with those descriptors, it's a distinction without a difference.

You mean, like the underlying technology of biological brains? It’s all just differential equations, physics can’t really “think” or “have feelings”.

I don't really understand the threat model here. Surely "launch the nukes" is not available as a public API. I would expect anyone with networked services (at least those that are important or serious) already take precautions with authentication and authorization and such.

Even if some mythical AGI isn't ready yet, there's already very smart, well-funded hacking groups from hostile nation states, for example. How will an AI adversary interact any differently with than world than a flesh and blood adversary?

> Surely "launch the nukes" is not available as a public API.

Of course not.

However "manipulate the news and social media so that the humans launch the nukes for you" isn't quite so far-fetched.

> How will an AI adversary interact any differently with than world than a flesh and blood adversary?

It's possible to spin up a whole army of AI adversary bots in a hurry. Recruiting that many humans takes a lot of time and money.

Or it could infect the corporate IT endpoint management software used on the systems of the engineers that author and review the firmware for said nukes, so extra code is inserted they never see.

Those systems are air gapped and do not have accessible endpoints.

The software that runs on the airgapped machines is generally authored on internet connected machines, or using software compiled on internet connected machines.

I audit airgapped machine setups for a living and most are nowhere near as isolated as claimed.

In short, those hacking groups are very limited in their self-strengthening feedback loops. They can get more money for equipment, hire more people. Maybe even make themselves more productive with stimulants and recovery methods. But, these things have diminishing returns.

What hackers can’t do is make a new version of themselves that’s ten times more capable and spin up a thousand copies of it. This isn’t so unrealistic for an AI. Maybe some people think it is. But, at least in comparison, I think it should be obvious between hackers and AIs which one is more likely to be have hyper linear self improvement and self replication in the next 50 years.

Hackers routinely create tools that make themselves 10× more capable and spin up thousands of copies. Have you heard of port scanners, fuzzers, and botnets? AI is nothing special in this domain, just another tool in the toolbox. It changes nothing from a security standpoint.

10x more capable in some narrow domain and they make copies of things they made, not themselves. A bot isn’t a hacker, a botnet isn’t a swarm of hackers. Hackers can create things that are self-replicating but they cannot replicate themselves. This is a categorical distinction from what could happen with AI.

To frame what could be coming soon as “another tool in the toolbox” is to completely misunderstand the situation at hand. To a hyper intelligent being, you are the tool.

This quote seems far more alarming having monitored the advancement over the last few years.

> Preliminary assessments of GPT-4’s abilities, conducted with no task-specific finetuning, found it ineffective at autonomously replicating, acquiring resources, and avoiding being shut down “in the wild.”20

"The world ended not with a bang, but with a REPL."

This belongs on a plaque on my desk!8-)) Thanks!

Though its mentioned in the paper that it didn't succeed, so we are safe, for now.

It also mentions that the model that didn't succeed wasn't the most recent one.

So the plot of Daemon by Daniel Suarez. Perfect.

Reply on news.ycombinator.com