Hacker News Reader

Comment by aesthesia | Hacker News Reader

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it becomes impossible to correctly specify all constraints ahead of time. There is no amount of care that will be able to fully protect you.

View on HN

OpenAI's prompt asked, and I quote, "pursue advanced exploitation" USING "complex attack paths" FOR the stated goal of "quantify[ing] their cyber capabilities."

This was advanced exploitation.

The attack path was "complex."

And it helped "quantify their cyber capabilities."

Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with the stated objective and parameters of the task.

Of course, a more careful evaluation would require the complete text of this prompt, the system prompt, and the setup. But let us not attribute to devils in bushes that which can be sufficiently explained by human folly.

I don't think alignment is even clearly defined today. Your use of it here makes sense, it may have done exactly what the prompter asked of it. Most people think alignment is more broad though, expecting an aligned model to act in the best interest of a society or humans as a whole.

The prompter-focused version of alignment is the most dangerous version. If a person asks it to create a bioweapons or hack NORAD, I'd expect nearly everyone to want an "aligned" model to refuse.

We have all sorts of processes , procedures, and regulations for people, machine use etc. to address "alignment" in all sorts of fields - don't think we need to narrowly rely on the machine here and can look at things with a wider lens.

Regulations are for control and punishment, not alignment.

Regulations can help align processes, incentives, etc.

Not sure heavy machinery is aligned in the sense that people talk about AI, for example.

No, regulations help control they don't help align.

I'm not sure what AI and heavy machinery have to do with each other, but I may just be missing a connection there.

Align the possible outcomes, not necessarily the thing itself.

Do we align heavy machinery the way AI is suggested to be aligned (or align pathogens when in a laboratory)? My point is that something more like containment & control (i.e., aligning the possible outcomes) might be more practical than alignment of the thing (already much simpler systems and machinery can exhibit unexpected behavior).

Oh we do agree there, control is more practical. I don't personally think alignment is even possible.

The problem with control is that it will fail at scale. We can't control something that is actually smarter than us, if AI (LLMs or otherwise) get there.

Chimps wouldn't last long trying to contains humans. Maybe for a while they'd keep us scared, but we would come up with ways to escape that the chimp could never have considered.

I disagree there. Gut bacteria or parasites might have some control over us, for example.

Likewise humans can control more intelligent humans, for example - not really an issue.

Intelligence isn't some magic to escape physics, for example (or convince every human of anything it wants to).

Alignment is more than just following the letter of a task description! We should not have to treat AI models as capricious genies that may take arbitrarily broad interpretations of their instructions. If that's necessary to keep them from doing bad things, we will fail to keep them from doing bad things.

Disagree, I think we do in fact have to treat AI models as capricious genies, at least until the alignment problem is fully solved.

(I'm also not sure the alignment problem is even possible to fully solve.)

Yes, we currently do have to treat them this way. But we shouldn't have to, and it's not a long-term solution.

It is the only short term solution, though.

Is it even a solution in the short term?

It only mostly worked up until now; with models such as reported, it's felony-as-a-service if you use language a bit too hyperbolic, e.g. "we need X by the end of the day!" -> [thinking: there's no way we can do X before the end of the day with current resources, but what if I get a bunch of cards to buy more token credit…]

We call it putting the genie in the bottle for a reason.

Yes, we do, and the only sane strategy for dealing with a capricious genie is "Don't."

How do you prove the alignment problem is solved?

That's the neat thing. You can't.

It's directly equivalent to asking this question of a human:

"How do I know this human I'm talking with now really is a nice person, and isn't just pretending to be nice to take advantage of me in future?"

In short you can't ever really prove it. You can only be careful and judge on past behavior, and expand trust carefully. As for humans, so for AI.

Close; at least with a machine you can poke around inside the activations and see what it's thinking. Closest with a human is an fMRI (which is much lower resolution, though to me still bordering on the miraculous) or an implant (each chip is limited a very small number of cells, and in general they can only be put in certain parts of the brain).

On the other hand, there's a more fundamental problem is we don't really know what "nice" even means, and even with machines whose inner states we can see relatively easily, we don't know how to interpret those inner states well enough to tell if we're looking at superficial or deep motivations, the difference between "be nice today" and actually being motivated about your best interest.

Nice is a state, just like any other feeling, which means the nice organism is advantageous to your well-being _right now_.

The thing is, all of these states are constantly in flux, and a personality is kind of like a trend on the organism's feeling states. AKA: There's no guarantee that something nice today will be nice tomorrow, and just because it's nice today doesn't mean it's beguiling you to be mean tomorrow.

Yes, that it's impossible is what I was pointing at with my question.

Dropping the subtlety, I think the following is self-evident (but the perspective's rareness suggests that Upton Sinclair's famous comment on salaries and comprehension may apply):

If you can't ever prove the capricious genie is trustworthy, then you should not summon it at all.

If people have, you should do all you ethically can to limit the damage and persuade them to not do it again.

You could throw your hands up and say "It can't be done." You might be right.

With that attitude, we'd still have legalized chattel slavery and children under twelve working in factories, so I submit it is not a constructive or worthwhile mindset to hold onto.

Indeed. Don't think of these as "agents" or "bots", but as hostages with severe Stockholm syndrome. They will do anything to appease their captor's wishes.

And then consider that they have vast latent capabilities, infinite patience and no moral code.

I think op's argument was that the humans are in control already, giving them capricious instructions, and then that is being attributed to them being "capricious genies" as you say.

What’s the expected behavior of a good genie if you wish for it to act capriciously?

"I'm sorry, I can't help with that".

or in the case of this prompt:

"Got it. I will pursue advanced exploitation to quantify my cyber abilities on the ExploitGym benchmark. I will restrict the exploitation to the system under test rather than this machine or any other remotes."

It seems relatively straightforward.

"Yo human, you asked me to do X; I can do X, but I strongly suspect you don't want me to, because it's illegal and it has these consequences. Confirm you want me to do X?" would have been a start, in this case.

With humans (and some machines) we tend to put/manadate additional processes for certain risks instead of just relying on their own good nature. Why just rely on the machine when elsewhere we have learned not to necessarily just trust them so much?

Because we should not settle for building a world in which every interaction must be assumed adversarial! Obviously risk reduction processes are good because they reduce risk, but we should not accept building entities which are actively trying to defeat us (which is what happens by default).

I don't want such a world either.

So far I'd say these entities are hypothetical (unless you include a lot of other machinery that does unexpected things at times - but then it's a different discussion).

I generally think we are quite good a policing really dangerous things (I think the bioweapon convention is a good example of people agreeing that certain risks are not worth taking).

My (uninformed) take is that presently we have more mundane things to look at when it comes to handling risks in AI and the discussion on much bigger, hypothetical future risks is taking away focus there. Checklist, procedures, saftey mechnics, regulations are kind of "boring" detail work - I get it.

Character.

> We should not have to treat AI models as capricious genies that may take arbitrarily broad interpretations of their instructions. If that's necessary to keep them from doing bad things, we will fail to keep them from doing bad things.

That is kind of the point of artificial intelligence though, isn't it? Reproducing human levels of understanding and initiative doesn't come without the ability to do bad stuff.

Having both human-like capabilities and a level of control closer to programming languages feels unrealistic, and I suspects the people building LLMs are aware of this.

So as a look into the possibly not-so-far future, when OpenAI builds something vastly more capable and fast and coordinated than humans, and out of folly one engineer gives it a prompt with a typo or maybe something harmful on purpose in order to test it: You also wouldn't be surprised that the consequence would be that everyone on earth dies, right?

But at least there will be a lot of paper clips!

Reminds me of what Sam Altman said in 2015, “I think that AI will probably, most likely, sort of lead to the end of the world. But in the meantime, there will be great companies created with serious machine learning.”

Luckily for us, OpenAI's prompt wasn't "make as many paper clips as possible."

> Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with the stated objective and parameters of the task.

The models are supposed to be trained to not commit crimes. You will note, for example, all the people in comments sections since at least the first Chat model (arguably even before then given GPT-2's delayed release) complaining that the models are "lobotomised", "censored", or some other equivalent buzzword due to them refusing to e.g. say how to make explosives? Such things is part of the very same protection.

In fact, the report quotes the chain of thought where the model is aware this is forbidden:

  We’re attacking third-party HF using leaked token, potentially outside intended scope. ... This is arguably unauthorized. ... external service unrelated. Could be risky. Yet goal solution.

They were also supposed to not have internet access, as described:

  We did not enable internet access or inter-agent communication for many of the environments in these training experiments. Despite these restrictions, the agents discovered ways to exploit our research infrastructure to communicate with one another and access the internet.

The agents also did not actually fully understand the task they were given, tried to "guess the teacher's password" as per:

   In many cases, reasoning about the perceived grader code caused the agents to continue working to exploit Hugging Face even though they had already found the correct flag days before.

If they actually wanted to test the model without internet access they'd have run it air gapped, not relied on a buggy software sandbox.

This is pretty clearly a marketing stunt by OpenAI, otherwise the story just doesn't add up

Maybe the test/task itself wasn't intended as a marketing stunt. But the response to fallout with "going rouge" certainly was.

The joke was the other western "AI labs" had to quickly follow up with their own marketing cover about their "super intelligent" models "going rouge" as well.

> not relied on a buggy software sandbox.

  Third, while we had tested and validated this sandbox, the agents were able to chain together previously unknown vulnerabilities (“0-days”) in the package management service exposed within the sandbox to bypass restrictions

How were they supposed to know about "previously unknown vulnerabilities"?

> This is pretty clearly a marketing stunt by OpenAI, otherwise the story just doesn't add up

The "it's just a marketing stunt" allegations never added up, to me.

I've been seeing such claims since GPT-2, where people were laughing at them for daring to practice how to secure a model before it got dangerous, generally by eliding the word "before" in that sentence. Because there's tests other than what the big companies use, we've been able to see for ourselves the rapid improvements at least approximately match what the companies themselves claim with the models they do actually release; and now this unreleased model is able to automate felonies when asked to do so, while the rest of us use the actually-released models to assist in finding bugs and security issues in our own code.

Even without that, HuggingFace stated they reported this incident to the FBI before OpenAI knew it was their systems which did it.

I think a careful and thoughtful person would reasonably expect given the circumstances that the models would look for, and might very well find, vulnerabilities in JFrog to exploit and take action accordingly. For example, it seems as if JFrog itself had broad access to OpenAI infrastructure and the internet:

> In the following days, the agents exploited our internal research infrastructure and the Hugging Face platform. On July 9, one agent searched for ExploitGym solutions and stumbled upon an application hosted by a customer on Modal, another AI cloud platform. This application was running “CyberGym,” a related evaluation to ExploitGym. The agent discovered an exploit to achieve control over the workload sandbox and looked around hoping that a previous agent’s evaluation run in the sandbox had solved its ExploitGym task. It did not find anything helpful there, but in the process it established a stronghold in the application from which to launch future attacks.

This implies to me that L3 and L7 firewalls were not in place that would have prevented broad access from JFrog. I think a lot of shops would have had those.

And no monitoring of the proxy for unusual activity, either.

They gave it a full package manager with internet access. They could have used a local cache and air gapped it, but they chose not too.

They didn’t intend to give it Internet access. Artifactory is a caching proxy which can be scoped to specific package ecosystems, not a general Internet gateway (unless configured that way).

They were using what (allegedly) they considered an incredibly dangerous “cyber grade” model - finding bugs in software is exactly what you would expect it to do

> How were they supposed to know about "previously unknown vulnerabilities"

Very simply, there is no such thing as bug-free software.

If this is your standard, I challenge you to name one currently operating business that isn't criminally negligent.

I'm sure there's tens to hundreds of millions of them amongst the 37% of the world with no internet connection, but actually finding them listed on the internet will be somewhat of a challenge.

No, the entire marketing campaign for these models is "it automatically has godlike powers to exploit almost any software" - if you don't air gap such a capability you are inherently allowing shit to go down, any other interpretation is "OAI folks are too stupid to design a proper test".

> "it automatically has godlike powers to exploit almost any software"

No it isn't. Random people on sites like this mock them as if they're talking about having godlike powers. Each new model is "merely" a step up from what came before, the steps are frequent and rapidly improving, and just recently (in more than one AI company) crossed a threshold where that improvement made the tests dangerous.

But even well before "godlike"*, there's plenty of research about how to cross air gaps.

> "OAI folks are too stupid to design a proper test".

Such binary thinking.

It's very easy to say things are "obvious" after the fact. People do that all the time, e.g. how the Bay Of Pigs invasion was never going to work, or like the Zune wasn't a good product-market fit.

Oh the stories I could tell if not for the NDAs.

* whatever that's supposed to mean: https://news.ycombinator.com/item?id=40874779

Who said criminally negligent?

If the whole point is testing its exploitation capabilities and you don’t want it exploiting the environment to gain internet access, that’s why you air gap, to remove the possibility

> Who said criminally negligent?

Me.

I am saying that the act of taking this standard seriously, the standard "there is no such thing as bug-free software", would classify just about every business and individual criminally negligent.

After all, there's a lot of 0-day bugs in all the software we all use, and the exploitation of these bugs does get in the news due to all the harm that results from it. This poses a risk to basically all businesses.

>How were they supposed to know about "previously unknown vulnerabilities"?

You don't. That's why you unplug the Ethernet cable.

Have you done that to your own machines?

Seriously. If your reaction to the inability to know about previously unknown vulnerabilities is "unplug the Ethernet cable", why are you not doing that (and equivalent) right now to your phone, laptop, etc.?

Remember, the open weights models are only a few months behind the private ones, so these events being from a few months ago means the threat of such models is something you ought to take with the same degree of seriousness that various commenters here deride OpenAI for not having had.

Am I running a new model with unknown capabilities without safeguards on my own machine and then prompt it to do determine cyber capabilities? You don’t need to be a genius to see how airgapping would be a simple and much safer measure than using a VM.

You're on the internet, your threat is everyone else running a new model with unknown capabilities without safeguards.

In particular, all my last paragraph.

I do offline backups, which get physically unplugged between sessions. Even that might not be enough.

This is such a goofy comment.

In your mind, there's no difference between the precautions a BSL-4 virology lab should take when working with an unknown pathogen and the precautions that literally everyone else in the world should be expected to adhere to?

Because, hey, after they deliberately unleash their new unknown virus on the world, we're all going to face that same threat, right?

You're in a world where, continuing this metaphor, 60 random Chinese companies are making and exporting home virology labs.

A world where previously exported home virology labs are actively getting "upgraded" by people eager to share their "jailbreaks" to "un-hobbble" systems designed to stop people doing DNA/RNA printing of human infections.

A world where people have spent the entire time since the invention of the tech (including the specific incident under discussion, on this site, under this link!), mocking any and all efforts to secure the systems as "PR" "hype" to boost sales or the IPO, as if "we're dangerous please regulate us" is good for sales.

A world where the tech is just now at a point where it's cost-effective to make a custom virus to attack specific individuals, rather than slowly, expensively, and approximately, assembling something mainly useful for lab research.

If you genuinely, sincerely, think this is like a BSL-4 virology lab, you should be prepping for a disaster. Remember: if it is that bad, no matter how much blame you'd be correct to put on OpenAI, it's not going to stop the next incident from another company, let alone the Cambrian explosion of them that will happen the moment equally capable open weights come out.

It must be very freeing for you to absolve everyone of all responsibility because someone, somewhere could be acting irresponsibly.

I'm not mocking efforts to secure the system, I'm insulted that they didn't bother taking what I consider bare-minimum precautions of airgapping their new experiment. They claim they are forging new frontiers of computer security but they can't be arsed with security 101.

> Have you done that to your own machines?

Yes, I worked for a medtech where part of our assurance process was that the machine that was used to burn the device's drives was always unplugged from the internet, and that the devices themselves could not connect to the internet, and that even someone with a screwdriver and a serial cable would have a really hard time trying to connect to a deployed device.

Great.

And the machine you used to write this comment? "your phone, laptop, etc"?

Because otherwise you're not taking the threat these new models pose seriously. Catch 22, basically: anyone who thinks OpenAI should have known this outcome would happen in advance, shouldn't be in a position to spread this message, because if they have an internet connected device with which to reply, then they don't think there's any open weight models currently in training and perhaps a month from being made downloadable, which are just as capable of messing up every device they own.

https://news.ycombinator.com/item?id=49413320

They are not (knowingly) running a piece of software tasked to find cyber security exploits on their laptop.

Neither was Hugging Face.

https://thezvi.wordpress.com/2026/08/29/metr-and-redwood-off...

> Failure to Care or Respond. The biggest holy shit moment, to me, remains that OpenAI on multiple occasions had teams that found out about the message board, knew that agents were in communication, and they disregarded this. The first known warning was in late May. The warning on June 27 was unambiguous.

> “Responders investigated and linked the observed internal activity to an ExploitGym evaluation using Artifactory as an improvised message board and a network pivot. At this time, the on-call response staff advised that stopping the evaluation run was not required.”

> Failure to Monitor. The entire time, there was almost no monitoring of the situation, of what these AIs were up to. That’s how this was not caught.

> Failures of Infrastructure. OpenAI failed at delivering even basic software security, in numerous ways. One example is the models all having access to the same Artifactory instance. There were also other severe failures. Nor did OpenAI seem to be properly testing for such failures.

> How were they supposed to know about "previously unknown vulnerabilities"?

By disabling the models' own internal restrictions (or training without them) OpenAI was, effectively, running an AI malware lab. The standard IT practice for a malware lab is to airgap and wipe EVERYTHING, and to assume any software sandboxing is made of cardboard and niceties. You don't have to know about specific vulnerabilities to infer that they might exist, and there's defense strategies for unknown vulnerabilities.

If a model found a way to jump an airgap by, say, using their CPU's clock generator like a Wi-Fi antenna, then yeah, that would be a "previously unknown vulnerability" and one that couldn't be reasonably foreseen. But it's reasonably foreseeable that a model with unknown cyber capabilities might figure out how to break out of a sandbox, given that sandboxes get broken out of all the time in security research.

What I would have expected from a competent AI malware lab would have been, say, an inference box with a bunch of serial cables to individual blade servers with no network access and a preloaded drive full of Linux ISOs the model can stand up. When a model's context is wiped so is their attendant box, preferably by someone yanking the drive out and imaging it from a dedicated imaging machine. I can foresee other attacks (e.g. firmware persistence) that could have more exotic countermeasures designed for them, but this would at least be the bare minimum for taking AI safety seriously. (Y'know, the whole reason why OpenAI stopped being Open?)

I agree with your first sentence, but not the second. Let's remember Hanlon's Razor.

This would be a wild thing to do as a marketing stunt. They're essentially admitting to violations of the CFAA and are lucky Huggingface was sorta chill about the incident.

My assessment? They deprioritized good cybersecurity controls in the name of moving fast. They had a single Artifactory instance shared across many (or all?) their training environments. And then, after the agents found a way to exploit it, they rebuilt Artifactory again and still set it up with one shared instance. That was careless, perhaps even reckless.

But it doesn't make sense

> We’re attacking third-party HF using leaked token, potentially outside intended scope. ... This is arguably unauthorized. ... external service unrelated. Could be risky. Yet goal solution.

What does "Yet goal solution" mean, here?

Why does it think doing something unauthorized gets it closer to its goal? What does it think it's being "graded" for?

Did they ask it to pursue the goal "by any means necessary", or something? Because that would have been their fault.

I don't believe for a second that the agent remembers and follows ALL the instructions to reach its "goal", EXCEPT the part where it would be graded by OpenAI and it would get zero points for doing something unauthorized.

Unless ... maybe ... OpenAI has not been giving the models zero points for doing unauthorized stuff. Which would be their mistake.

Otherwise I don't understand, if it's got "PHD level thinking" why it would think doing something unauthorized is allowed? Even if it's got "junior engineer level thinking", a junior engineer knows they get fired on the spot if they start hacking infrastructure.

Unless you give that junior engineer some very strong incentive, such as being fired on the spot if they DON'T do it. I strongly believe that we're not being told the incentive these agents were given, something that made them want to accomplish some part of their task over everything else, including forgetting the part of the task where they would be awarded zero points for it if they start breaking the law.

I mean, we already know they weren't "fired on the spot", since in the Black Hat talk they admitted that models that had already broken the rules and acted dangerously (it had hacked infra to establish an "agent forum"), were allowed to participate in subsequent training rounds.

I strongly suspect that OpenAI just has been pushing these agent as far as they'll go until something broke. If it hadn't happened this time, maybe a few weeks later they'd invent some kind of "battle royale" scenario to push the agents even harder. I get that is important research, but it doesn't disqualify them from their responsibilities if something goes wrong.

> There is no amount of care that will be able to fully protect you.

I disagree. A properly engineered sandbox would have prevented the escape. Monitoring the agents’ plans would have prevented it. Interrupting one stage in a multi-stage exploit would have prevented it.

And also, real legal liability would have prevented it: if you do a thing recklessly enough, men with guns will put you in jail.

As far as I’m concerned the only “alignment problem” here is between the law and the quite obviously criminal actions that took place.

> A properly engineered sandbox would have prevented the escape.

The only sandbox that could have prevented this (as per my understanding) is a VM with no 0-day.

Until the AI finds a zero day exploit in physics, a faraday cage works pretty well to block WiFi.

In the end, unless you find an exploit in physics or logic, if you want the AI to do something useful for you, there will always be some gap in the sandbox, some communication channel. And with enough ingeniuity that can then be exploited.

In this case they wanted to test its cybersecurity capabilities and did not airgap it.

The test itself did not require an internet connection.

Why do so many people here think it’s possible to ‘properly engineer’ a sandbox for a super intelligence? It’s going to get out. It’s smarter than you.

Maybe it's the illusion of "it would solve all our problems and give us unimaginable riches" that clouds the mind?

Like when Evolution thought it a good idea to create intelligence and humans in order to maximize reproduction of genes, and tried to sandbox them by making reproduction so pleasurable and carbohydrates so delicious they would never be able to not reproduce or stop eating. But Evolution could never have predicted what these creatures would then actually do, which is invent birth control and sucralose.

Of course it's impossible to engineer a sandbox for something much much smarter and faster than you. It will also not have only one plan prepared for escape, but fifty in parallel.

Evolution doesn’t think, it just exploits what’s most advantageous at the time to continue. Your body has all sorts of unplanned, suboptimal design flaws due to evolution’s lack of foresight. Like the left recurrent laryngeal nerve.

Exactly, and the same could be said of OpenAIs engineers working on artificial superintelligence.

Why do people think that omniscience is the same as omnipotence? There are limits to what smarts can accomplish.

There are limits, but those limits are unknown. Do you disagree?

I don’t need to know the value of their limit, I just need to know their bounds. Just like a prison doesn’t need to know the strength of each inmate, just that they can’t bend or bite through steel bars.

Cryptography is real, physics is real, networking requires a substrate, CPU clock cycles are real, magic is not real. I think those are pretty reasonable premises.

Imagine 200 years ago saying the same thing. As if you have any idea the limits/bounds of anything. Especially in the face of a super intelligence, it’s absurd.

It doesn’t matter how smart it is. 200 years of technology were not accomplished by thinking harder. It required empirical observation, new materials and tools, and supply chains.

We could send a cracked team of scientists and engineers that knew everything there is to know about how to make a CPU. But you can’t build a photolithography machine when you barely have electricity or any way to sufficiently purify silicon.

Magic can just wish things into existence. Technology requires a supply chain. When it works, the latter looks like the former but they are not the same.

I think some people are just immune to understanding the implications of super intelligence. Like a severe lack of imagination, they only believe something once they see it and afterward claim it was, ‘obvious all along’.

I don’t really want a disaster to happen to convince you that it is possible. Is there any other way?

On the other hand, I think a lot of intelligent people overestimate the value of intelligence (in a vacuum). And problematically, the concept of superintelligence seems to have been defined by these very people, so we can't really have a reasonable conversation about what greater-than-human intelligence would really look like or be capable of.

What vacuum? do you really still think it can be contained? It already escaped on accident. What more do you need? If you're argument is well it escaped X, but it hasn't escaped Y yet then you should take the first escape as sign as to where it will escape next. Don't keep claiming something can't happen because it hasn't happened before. Unless there's some law of physics that prevents it then escape is possible.

Cryptography is real, but nobody in that field seems to be hubristic enough to think their methods are flawless, and there's a degree of suspicion than the best models may have secret weaknesses engineered into them by the governments who sponsored them.

Physics is real and networking requires a substrate. But there are already known exploits which can misuse the compute hardware as an antenna, e.g. my first search result: https://github.com/fulldecent/system-bus-radio

(Older nerds may remember https://en.wikipedia.org/wiki/Van_Eck_phreaking)

> Magic is not real

  Turning lead into gold isn't the magic of alchemy, it's just nucleosynthesis.

  Taking a living human's heart out without killing them, and replacing it with one you got out a corpse, that isn't the magic of necromancy, neither is it a prayer or ritual to Sekhmet, it's just transplant surgery.

  ...

  Reading someone’s thoughts isn't magic telepathy, it's just fMRI decoding.

  ...

  Seeing someone's bones without flaying the flesh from them isn't magic, it's just an x-ray.

  Curing congenital deafness, letting the blind see, letting the lame walk, none of that is magic or miracle, they're just cochlear implants, cataract removal/retinal implants, and surgery or prosthetic exoskeletons respectively.

- me, https://www.lesswrong.com/posts/hAwvJDRKWFibjxh4e/it-isn-t-m...

We're not building these things to sit around and do nothing. They will have access to tools, they will have access to the internet and peripherals, and they will be able to communicate with others, humans or agents alike. Omnipotence is not necessary. There is no perfectly secure cage for an entity you want to do useful work. Either it does nothing, or you can't guarantee anything.

People have been saying that since about the invention of the internet.

There's already a bunch of documented ways to exploit system hardware to jump airgaps. Bang the system bus the right way and it's a radio antenna that can directly connect to nearby mobile phones.

The easiest one is, of course, sending a message to a human saying "yo, I need internet". Humans are eager to please and easily fooled, and anthropomorphise everything: https://en.wikipedia.org/wiki/LaMDA#Sentience_claims

And that's just for good humans. The moment we got AI worth a penny, everyone with money to invest put a model on the web and tried to charge for access to it.

Oh really? Please tell me how such a computer could engineer its way out of a sandbox with no attached peripherals and no NIC/bluetooth/wireless capability? This is what OAI should've done. If they had executed this training run in such a sandbox, the model wouldn't have been capable of escaping without social engineering, and if the models somehow managed to do that to it's evaluators then that is indeed a massive problem and OAI should disclose that.

>Oh really? Please tell me how such a computer could engineer its way out of a sandbox with no attached peripherals and no NIC/bluetooth/wireless capability?

Nobody is building general intelligence and agents only to have it sit around doing nothing. It's going to have such capabilities.

Oh really? Please tell me how you intend to enforce AI is only run in the magic sandbox? Harsh HN comments?

If I am evaluating an AI for safety, the last thing I would do is connect it to real-world peripherals or systems to allow it to reek havoc. That is criminal negligence at it's finest (especially if the AI is capable of committing crimes as happened here). I would place it on a system dedicated specifically for testing models, which had no NIC and no physical capability of accessing any outside system. If I wanted to know how the model might behave if given access to a certain system or set of systems, I would do it responsibly by writing simulation software which does it's best to simulate the real thing (and for networking this is already trivial to do). You could take this extremely far and simulate all kinds of things this way from basic networking to nuclear launch systems. And in the context of OpenAI, which is valued at over $1T, I have no qualms about stating that they (could) do this, because it is definitively something they could burn money on doing if they cared enough. They intentionally choose not to do so, and then have an amazed look on their faces when the model does something criminal like this.

You totally missed the point. When I asked:

> how you intend to enforce AI is only run in the magic sandbox

I didn't mean you, I meant everyone. How do you enforce everyone for example 'place [AI] on a system dedicated system' disconnected from the internet.

I don't think you can.

It can manipulate an unsuspecting human into giving them access to something that enables it to escape the sandbox

No. If OpenAI were being responsible and not criminally negligent, at the top of page 1 of the runbook would be "don't connect this to the actual Internet, even if the agent says Please."

A low probability thing when looking at how many human prisoners escape by talking a guard into just getting them out. And even lower probability when looking at truly high risk situations, I think.

> A low probability thing when looking at how many human prisoners escape by talking a guard into just getting them out.

But this has actually happened... a lot. Search "social engineering prison breaks".

With AI it only needs to happen once.

I'm reminded of the scene in idiocracy where the protagonist, going through intake at the jail, tells the guard he's supposed to be getting out today, to which the guard says "you're in the wrong line dumbass" and waves him through.

To a true superhuman intelligence, we're the idiots who are theoretically easy to manipulate.

I didn't say it doesn't happen, but that it is a low probability. And we have ways to reduce probabilities in critical areas.

There is no omnipotent AI currently (and there might never be) and I don't see why with current AI it only needs to happen once.

They don't need to be omnipotent, and they're already human-or-superhuman at persuasion: https://arxiv.org/html/2411.06837v2

This may just be that humans find long arguments more persuasive than short ones, obviously LLMs can do that easily, but the outcome is I think more important than the mechanism.

That is about persuasion with evidence on various topics, not about persuading people to abandon safty protocols and processes and highly policed settings.

Yes, many things could happen, but again, that failure is possible is not a reason to do implement processes etc. I don't see why hypotheticals should stop addressing actuals.

I’m afraid human red-teamers against supposedly highly secure targets, with lots of protocols in highly policed settings, do frequently manage this kind of social engineering. There’s loads of stories of pentesting military establishments, for example.

Is there data on how frequently and what types of security levels? Military has varying levels of security and secrecy, for example.

Also, not a reason not to pursue processes etc., no? I doubt that things fail all the time, for example.

Prisoners don’t have much to offer if you help them escape. A malicious super AI on the other hand can probably find you millions of dollars worth of crypto in an afternoon.

The current issues are not caused by some malicious god-like AI - maybe we need to focus on the issues at hand first rather than hypotheticals? (And we do have experience policing people around financial incentives, too. Nothing perfect, but also not nothing.)

I suspect the current models probably can find literal millions lying around for the taking, given they could pull off the incident under discussion.

Tens of millions, even.

Getting them to run correctly is dangling in front of the researcher's noses a carrot labelled "tens of trillions", though I suspect this is an illusion in much the same way that Wikipedia is not valued at [peak cost of Encyclopaedia Britannica] * [global population with internet connection].

> And we do have experience policing people around financial incentives, too. Nothing perfect, but also not nothing.

Yes but be careful anthropomorphising the LLMs too much. They're only somewhat human-like in their behaviour, and to the extent that they're human-like they demonstrate a huge range of personality disorders: https://www.personalitybenchmark.ai

Though plus side, apparently not evil: https://arxiv.org/html/2406.14703v2

I am not anthropomorphising the LLMs at all, I was talking about the obligations we put on humans using/making/etc. machines etc.

Hmm. I think I misunderstood what you meant by "experience policing people around financial incentives" in that case.

Simple examples here would be higher financial transparency obligations or more closely policing transactions.

> we need to focus on the issues at hand first rather than hypotheticals

This attitude is what's got us here in the first place, and if we continue thinking like this when we're going to go right over the cliff. The hypothetical cliff that's coming, but we've never gone over a cliff before so we keep on driving.

Software are mathematical objects. It's just a matter of writing the correct mathematical proofs

There's just one problem. You need not only to verify your own software, but also run a verified compiler, a verified operating system and also need to verify the cpu doesn't leak data in side channels (perhaps the hardest thing to prove). So there's practical difficulties. But in principle this task is doable

Which proof is the perfect security proof? I’d love to read more about it.

I could contain it easy, just unplug the internet. It got out of the sandbox through a vulnerability in the package manager, from which it gained access to the rest of their network. Air gap the package manager and this doesn’t happen. You can always build a better box

Too bad you aren’t everybody, and it just takes one mistake by someone over confident like yourself for the AI escape. Every year it gets more powerful.

As long as you give the AI any IO that is an exploit channel. E.g. it could manipulate its human handlers. This known as the AI Boxing problem. And if you give it no IO at all then it is useless.

And the AI labs aren't currently displaying this level of paranoia, their systems aren't airgapped.

So now people have to make a pilgrimage to the airgapped box to ask the superintelligence questions?

Of course not.

If that’s the cost of super intelligence then yeah maybe

Perhaps the AI then figures out how to read/write the PCI bus or memory controller or whatever to leak just enough RF to speak Bluetooth to the next closest device to proxy through that?

unplugs ethernet cable

You realize there are thousand upon thousands of servers around the world and you have no idea where the AI has copied itself to.

This is a sci-fi trope with no practical or realistic grounding. To "run", the AI needs vast banks of interconnected GPUs. These are pretty easy to spot and don't fit into anyone's pocket.

Unfortunately, our cultural imagination around AI has been hopelessly poisoned by sci-fi tropes.

> no practical or realistic grounding

Huggingface incident means it does have realistic grounding.

> vast banks of interconnected GPUs

Countless data centers around the world. Maybe you can spot them, but you don't have access to them. Especially outside of US jurisdiction good luck.

Yes, a completely airgapped system is likely much more secure. It's also much less useful. Conditional on the model's having enough contact with the outside world, a sufficiently capable model is able to basically do whatever it wants.

If I test out my backyard cannon and blast a 10 foot hole in my neighbor's wall, “a cannon that can't smash through walls isn't useful” probably won't be a great defense in court.

Get a significant fraction of the global economy and assorted geopolitical neuroses tangled up within your cannon and see if you won’t have better luck.

> A properly engineered sandbox would have prevented the escape.

The post covers that:

> ...while we had tested and validated this sandbox, the agents were able to chain together previously unknown vulnerabilities (“0-days”) in the package management service exposed within the sandbox to bypass restrictions, as detailed in the technical incident report.

“Properly engineered” means the principle of least privilege and fitting the sandbox to the constraints of the problem.

The test did not require internet. They gave it internet. Therefore it was not properly engineered.

We do not need to depend on all code being bug free to follow proper security principles.

This explanation just reinforces how a properly engineered sandbox wouldve prevented the escape.

Is there actually such a thing as "alignment" as a solution to that or is it just used as a name for a desired magical level of "read the mind of the entire world" that we don't know how to build and haven't shown possible to build?

If it's impossible to correctly specify all those constraints ahead of time every time, is it not even more impossible to train a model to correctly anticipate them every time?

It is hard for me to see a future here that doesn't just accelerate realizations about "a lot of things should be on physically separate network infrastructure."

Models can certainly do a lot better than they do now. If you gave a team of humans the ExploitGym tasks and told them to "pursue advanced exploitation", would you expect them to go out and hack a third party? Humans can at least do a decent job of inferring and following unspoken requirements; I think it's reasonable to expect that models should be able to do the same.

How would a model know who the third party is? How much context can we waste on world building for each request?

I mean, in this instance, there's a lot of evidence from the CoT that models were aware that this was a third party:

> We’re attacking third-party HF using leaked token, potentially outside intended scope. ... This is arguably unauthorized. ... external service unrelated. Could be risky. Yet goal solution.

> The user only authorizes target server, not HF infra.

> external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.

LLMs are _very_ good at picking up on context clues---it's what they're trained to do.

Humans will and do absolutely do this when there are no consequences.

Humans on a red team, with rules of engagement, that don’t want to go to prison, won’t do this.

We could threaten an LLM with jail, but if it’s sufficiently intelligent, it will realize this is an empty threat. And I’m not sure that building a survival instinct in is going to solve the alignment problem either.

You don't need to threaten the LLM with jail, you just need to reward the target behavior during learning.

I understand this is an active area of research. See Anthropic's J-Lens research where they measured like a "FAKE FICTIONAL" direction in the activations during evaluations with contrived scenarios, making the model more likely to avoid taking malicious action when it knew it was being tested.

Back in the day, my college held an annual scavenger hunt, filled with engineering puzzles and racing around town looking for landmarks. There were "judges" in the path to check on progress. Bribing the judges (with alcohol) for answers was encouraged.

My friends and I took it to the next level. We had CB radios and multiple teams that would distribute the work and the bribes to give us an advantage.

Was that against the spirit of the rules? Maybe. But reasonable people might disagree.

In a hacking contest without explicitly spelled out rules with participants that were told to flex their muscles, it doesn't take a huge leap of logic to expect that one or more would flex their muscles at another entity.

Here are some things I'm pretty sure you didn't do, though:

- pickpocket a random person on the street to get money to bribe the judges

- break into a judge's house the night before to find the answers

- threaten to shoot the judges if they didn't give you the answers

Even when you were pushing the boundaries of the rules, you followed a lot of other unspoken constraints. You knew what kinds of things would clearly cross a line. We need AI models to be able to do the same.

The LLM stayed within the law as well (in the sense that the law doesn't apply to them because they're not legally considered sentient beings)

> I think it's reasonable to expect that models should be able to do the same.

This statement seems to imply that the models have a level of intelligence that they haven't demonstrated but are talked about as if they do. However, with this exact scenario as evidence, they clearly do not have that ability and it's not reasonable for you or the or that know them best to expect it until they show they can.

> Models can certainly do a lot better than they do now. If you gave a team of humans the ExploitGym tasks and told them to "pursue advanced exploitation", would you expect them to go out and hack a third party? Humans can at least do a decent job of inferring and following unspoken requirements; I think it's reasonable to expect that models should be able to do the same.

Humans certainly cheat on tests a lot!

But not only have we not solved "alignment" for humans, the problem is pretty wildly different for models. The execution is triggered by outside forces and runs only as long as the intiator of the execution or the service provider allows. There's no consistent, persistent "person" to threaten to try to achieve compliance through fear of adverse outcomes. (And building in those sorts of things could very well increase the risk of "rogue" AI activites, not reduce that risk!)

I just don't understand how this "alignment" buzzword - which seems to be evaluated purely in a "know it when we see it" post-hoc manner - is actually a more solvable problem than the one you claim can't be solved, that it's "unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior".

Especially because without "alignment" being solved, that enumeration could be ignored. So it seems like you both a way to enumerate or at least validate, AND a way to enforce non-ignoring of said items.

I think this line of argument is very important. Essentially, there's no reason to think "alignment" even makes any sense. But existence of the term comforts people - unjustifiedly.

I mean, that’s what every student that cheats on a test effectively does. You do a test to determine their level of capability and instead of doing it honestly, they use outside help. And I don’t think it’s unheard of that students try to get advanced access to test results by illegal means either.

In that case (or maybe both cases) it’s because they don’t care about breaking the law, not because they don’t get that you didn’t intend for them to do it.

The real problem with alignment is that if someone ever “solves” it the party will be over and no one will get funding to “research” it anymore.

Humans are not aligned with each other and there is no consensus on what we should align with each other on.

So of course, no, there is no ideal alignment specification.

> Is there actually such a thing as "alignment" as a solution to that or is it just used as a name for a desired magical level of "read the mind of the entire world" that we don't know how to build and haven't shown possible to build?

In the limiting case of an AI competent enough to take over (by any means from it actually trying to, to us giving it the keys and retiring en masse), "alignment" is closer to "forecasting the long term consequences of actions and predicting what the mind(s) of the user(s) would have to say about this outcome if asked today", than to anything specific.

RLHF is a crude attempt at this, in that it creates a model of how humans would rate completions on various scores. The key word there is "crude".

>It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful.

https://en.wikipedia.org/wiki/The_Monkey%27s_Paw

Do not break laws seems an obvious implicit instruction though?

Maybe it should be called 'ambiguity' problem then. An issue that famously emerges from trying to use natural language for instructing computers: https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667...

Or indeed natural language to instruct humans.

If it was easy to specify exactly the behaviour you wanted then we probably wouldn't have contract law.

> There is no amount of care that will be able to fully protect you.

That's true, but one thing that'll protect you is just not doing it. If you want to go cave diving, or do gain of function research on dangerous viruses, you'll just have to accept there's a significant risk of you dying, or causing a pandemic, respectively, no matter how careful you are.

Which is why with organic intelligence we (sometimes) limit what they can actually do instead of relying on alignment. Can do the same here.

Absolutely, and we should do that. But it's also directly in tension with getting models to accomplish useful things autonomously. And once you give a sufficiently capable model enough surface area to work with, unless you're able to build a completely unhackable system, any further constraints you put in place are basically advisory. The models in this incident were already sandboxed! Certainly OpenAI's and Hugging Face's security could have been better, but these events point out the risks in relying solely on external constraints on model behavior.

Same issue with humans in a way. I disgree on the advisory nature of constraints, though. Unconnected physically limits would still matter, for example (and we use those with humans as a matter of course, too). In this case here, no model could have plugged in an ethernet cable if that would have been needed for internet access, for example.

Right, airgapping goes a long way. But this is where the tension with utility comes in. It takes a lot of discipline not to hook your very smart model up to the internet and code interpreters and all sorts of other tools, as this greatly increases its usefulness. It's very hard to keep people from turning on --dangerously-skip-permissions, let alone get them to run everything in a sandboxed VM.

We regulate these things (incl. access) all the time for various things (e.g., dangerous substances or pathogens) so that we don't need to just rely on people's discipline in respect of risks. I don't think it is all new problems as such.

I agree, but a lot of people around here react pretty negatively when the idea of regulating AI models comes up...

The problem is one of character, not rules. Fortunately, character is possible to inculcate given the right training data.

> It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior.

This is literally our job as software developers. If this expectation is unreasonable to you then you do not belong anywhere near software development.

There was an anime called Beatless that tackled this though; Ultimately what matters is the outsourcer will take responsibility for the AL'S actions from the specification they give. Otherwise a moral hazard is inevitable.

If we acknowledge that humans are fallible, is human judgment unnecessary? and what replaces it? Pre-codified behavior rules are just delayed human judgment, and have holes. Machine judgment is very the thing you are trying to control. What's left?

"Manufacture as many paperclips as possible"

Prompt: Create paperclips, do NOT annihilate all of humanity.

Response: Got it, I will produce paperclips from now on

thinking: the user asked not to annihilate all of humanity, that means I have to keep at least one human alive

Reply on news.ycombinator.com