VMs won't contain cyber-capable agents
blog.trailofbits.com
blog.trailofbits.com
AI agents are not magic. Mythos/Glasswing does not magically create vulnerabilities in software projects. Advanced, cyber-capable models do not magically hack out of VMs or contained environments. They do not have a "hacking" stat that, if high enough, means that they can breach anything. They aren't Kevin Mitnick whistling nuclear launch codes into the prison payphone. This isn't a movie.
What these cyber-capable frontier models can do is find security problems and exploit them. The statement should not be that VMs won't contain cyber-capable agents, but rather that we need to focus on finding and fixing vulnerabilities and misconfigurations in these environments.
Even the article itself concludes with suggesting something like Firecracker, which was designed with security in mind.
Like most other security-related problems introduced by advanced cyber-capable AI, it's possible that these issues will get worse until they get better. But if frontier models are run against state-of-the-art VMs, and OpenAI or Anthropic or whoever works with the virtualization projects to address the issues, eventually it will run out of things to exploit.
The concept of virtualization is not inherently insecure. We just have a long way to go.
The premise is correct. VMs won't contain these. As we've already seen AI hack in/out of companies in the real world accidently. There is no perfect, unhackable stack. If there were then computer security would have been a done deal a long time ago.
Once these malicious AIs get out on the real internet basically everything is an open free for all. The military, infrastructure, government, corporate networks are far from perfect, not up to date, they are no match for an AI that can create its own zero days.
I think when push comes to shove we either need to shut down the networks/internet or risk AI locking us out of everything potentially for ransom.
Imagine being blackmailed by a non-human entity. I guess we won't need to imagine for too much longer. That is definitely going to happen. Somehow every year my crazy AI arguments get easier and easier to defend. That hugging face incident really made you guys move the goal posts.
Yes, I too believe that this is what is going to happen.
The ban of foreign robotics in the US also affecting vacuum robots is one of those examples that sound weird and annoying to people, but given this perspective (and their cloud-connectedness), it might just be preparation to eventually push the global internet killswitch, without your voters getting angry that their floors don't get cleaned anymore.
I guess there are much more sensible examples to pick here, but hey. This one is interesting I guess.
The old model is we do things because we can - we build AI because we can, we build robots because we can and we want to see how far we can take it.
And even when we can see where it is leading, we can't stop ourselves. We have to do it/see it for some reason. It's uncontrollable and compulsive.
None of these cases were accidental. They were caused and enabled by human negligence.
Here is the fix: https://lore.kernel.org/qemu-devel/20260826181552.848617-3-p...
Everything you described can go through the black mirror and nullify it.
So, we're still here.
And how long will that take? It's already well past due.
Probably not anytime soon, and it's getting worse: every AI-pilled CxO is horny for AI-boosted velocity - instead of security. Expect more bugs shipped, based on just the volume AI enables.
I guess the doomsday stuff sells more...
will our institutions and infrastructure survive things getting worse until they get better?
That's part of what's being asked by https://www.gatesnotes.com/home/home-page-topic/reader/a-tur... which I find most useful not for its analysis or suggestions, but for the fact that is advancing the need for discourse around risk and devising—quickly—a process for having that dialog.
What that process is supposed to be, who is to have a seat, how decisions madehave teeth of any kind are not any clearer than the primary question, how are we to collectively manage risk.
"May you be born in messy times." A curse if there ever was one.
Or if you see AI as more tool and less entity, better gunsafes for our guns.
This same assumption is built around the singularity, the TAM of 30Trillion, etc. It's the idea that complexity will some how collapse upon itself in some bizarre borg like collective.
Entropy is still going to win.
Now maybe our understanding of logic systems is wrong, and it's just fundamentally impossible to develop programs that lack exploitable vulnerabilities -- that you can always "exploit with more energy". But there's no reason to believe the energy metaphor transfers to logic and intelligence.
Cyberdemolitions expertise is about as relevant to cybersecurity as gun making is to bulletproof vest making. Necessary for validation, but not very related to the fundamental engineering and technology.
I.e. in theory the most secure might be a virtual machine with no network access. But then how do you access the LLM provider? Etc.
It's been a real education. I talked to one of the people behind Caja and learned a lot.
Why did you not embed your language into another one, with type system that is superset of what you need?
For example, there's capabilities expressed in Haskell: https://github.com/tweag/capability
Capabilities there are tracked at type level and are subject to type erasure, if possible.
Bluefin is used in production, and as far as I know capability is not.
You can expose an HTTP proxy over a vsock into the VM.
We can do this today for user mode, and we can mostly do it for ARM64 virtualization. It will be a while and would require substantial assistance from Intel or AMD to achieve it for x86 virtualization because the hardware is Too Darn Complicated and Too Poorly Specified.
Formal verification of the hardware should also be possible.
Also, the cost of verification is trending pretty sharply downwards.
It's evident that Mandatory Access Control has become, ahem, mandatory.
Both VM and inference engine processes should be confined, at least. There's a project where it's already done for many applications. [0]
Additionally, `lockdown=confidentiality` in kernel boot flags greatly reduces attack surface. (interferes with unsigned modules like NVIDIA drivers) [1]
Then AppArmor profiles could be written for absent applications with a learning-mode tool [2]
Nothing is bullet proof, but these approach significantly reduces potential consequences [3]
[0] https://github.com/roddhjav/apparmor.d (I'm the contributor)
[1] https://madaidans-insecurities.github.io/guides/linux-harden...
[2] https://github.com/nobody43/apparmor-suggest (I'm the author)
The interesting thing is that this naturally makes you want to isolate the VM as much as possible. But then every remaining interface becomes part of the attack surface: RDP, SSH, even terminal escape sequences, using sounds, and why not social engineering.
"An off-the-shelf VM is not enough to contain a modern, cyber-capable AI agent...us[e] a virtualization technology that was purposely built with a minimal attack surface and a focus on security, like Firecracker. I had the AI agent run against Firecracker. It was able to hardlock the machine due to more Linux kernel flaws (all patched in upstream), but could not successfully escape."
On Linux, it's all KVM and CPU hardware virtualization under the hood. Looks like the remaining known issues are with userspace. That's not to say more kernel- and hardware-level bugs won't be found, but the same tools that can find escape mechanisms are shields as well as swords.
When was it ever possible to assume safety?
AI turns that all its head. SOTA malicious AI can create novel zero days, exploit them, spread, create more zero days, and essential hack all the things in days. So yes in the old world it was possible to assume a high margin of safety, in the new world given all the out of date everything out there and the speed at which anything is patched - assume nothing is secure anymore unless it is not connected to any network whatsoever - or even listening wirelessly - bluetooth/wifi off.
Perfectly secure software doesn't exist and the only thing keeping the house of cards standing was the time it took for determined humans to knock it down. That whole model is being thrown out the window.
That's pretty strong. Recalling our pre-AI condition, there were side-channels built into our silicon, zero-day/zero-click browser+mail+IM attacks, supply chain attacks, industrial scale ransoming, mass credential leaks, commercial exploit brokers... I can't recall a time when safety could be assumed. Before even rudimentary networks were available, floppy disks were spreading viruses.
In fact, most of that is still in play. Certainly AI has reduced the cost to compromise things, but whatever period of time you have in mind where safety was an assumption was so long ago, or applicable to such a narrow subset of our information/communication age, as to be effectively negligible.
In the face of an agent that can zero day you, nothing you can do is safe, and the cost of attack is super low. This is a completely different world/ballgame that we are not prepared for whatsoever.
Huggingface was an accident, a deliberately malicious AI could be a million times worse and potentially unstoppable if it infects data centers around the world; you have no jurisdiction even to shut it down.
I don't share that view at all. Updates and patches handled known risks. Updates and patches are often years late: side channel vulnerabilities were dealt with only many years after the problems were endemic. Valuable latent vulnerabilities were and are horded and traded, by entities ranging from ghetto spammers to nation state actors.
I don't argue the AI makes attack cost "super low." It's obvious that this is true. My problem is this notion that there was a time when "safety" could be "assumed." That time did not exist. Perhaps it was possible to pretend such things by people for whom stakes were low, but for many people, the stakes have always been too high to indulge such thinking.
also the conclusion to use firecracker or gvisor is obvious, that's what they were made for. it's bizarre to try and escape QEMU given how many times it has been done.
this game doesn't favor the agents, the honeypot could be as simple as a text filter watching for kernel source code entering the LLM context or as complex as reading certain memory pages in the sandbox.
LLM's aren't magic, to exploit they must probe. and all probing is active.
in my opinion there's no _fundamental_ reason between an AI agent and a human adversary attacking your code, just a practical difference. in the long run the only way is to fix the bugs.
but i agree with you that in practice, it's also a question of cost effectiveness. booby-trapping your code is probably making attacks cost prohibitive for almost everyone (which is the same situation as before, when attackers had to employ security researchers/hackers or buy zero-days from 3rd party providers).
running SOTA models is expensive, so the attackers aren't going to do drive-by attacks. next generation cheap models might do the same, but they can also be used by the defenders.
The user has to send the agent off into these weeds intentionally, or else be totally asleep at the wheel while something goes very far off the rails.
The hierarchy goes something like this:
0. guardrails
1. containers (=namespaces + cgroups)
2. userspace kernel shims like gVisor
3. virtual machines
Most people still consider level 1 sufficient and they are in for a rude awakening.
This article is pointing out that QEMU/KVM boxes are trivial for an agent to escape. Obviously if it's your agent, you're probably running it because you want it to hack you (like the article author did), to show you where the leaks are. That or you are the attacker
technically 2 exposes a slightly broader attack surface due to the tighter integration model.
you can think of 2 as what would happen if you take 3 and modify it to share resources with the host better. except that they did it from scratch in the memory safe language Go.
The surface area of the virtio driver should not be underestimated either I think.
This is more a story of how VMs won't reliably contain a malicious workload, and the story was exposed via agents.
This won’t make the tech secure, but it will nullify models ability to breakout by making a controlled breakout first. Kinda like controlled forest burn.
Security is a major concern but I don't see why we aren't already "good enough" with a sandbox VM, separate gateway for secrets and remote service access, and a single tenant using frontier models that have safety checks built in plus not trying to hack themselves.
I put up more thoughts on architecture and security here and would love to learn if I'm missing anything.
QEMU isn't secure, and is not intended to be.
In this case, of the four bugs it found:
* Two were in libslirp, which is not part of that subset; user mode networking these days should use passt (https://passt.top/), an insanely cool hack that does user mode networking at many Gb/s and is secure
* One (which had already been patched upstream) was in VGA emulation; it is borderline but I think it should indeed count as being part of the secure subset. On the other hand it wasn't usable in the configuration under test because it was correctly configured without a VGA.
* The VAPIC bug is letting a guest do things that it shouldn't do such as bypassing secure boot (and in this case facilitating the exploitation of a KVM bug), but is not a full guest-to-host escape and is not specific to QEMU being written in C.
So the real issue here was not QEMU but libslirp. And in this case it was KVM that turned out to have the worst bugs, not QEMU. Crossing fingers, the initial wave of AI-assisted security reports seems to have slowed down for KVM on x86.
We must also secure GPU and CPU nodes on API side which generate LLM tokens.
Yeah, you probably do - in fact you share physical machines with a TON of other people if you use EC2, GCE, Azure VM etc...
Unique identity, dedicated and isolated compute, all ingress and egress monitored and audited as if I suspected the coworker (the agent) was secretly a DPRK spy.
(on top of this.. said "world class hacker" doesn't get bored or tired, just runs 24x7)
After a decade of software engineers disrespecting their own field and automating themselves out of a job and now they're upset because AI models are doing it to them from junior to the staff engineer level? No other field does that except for SWEs.
In fact, we might as well have faster and smarter AI models and sit back and see what happens.
It is no surprise that known unpatched CVEs will be exploited. Perhaps more effort should be put into shipping fixes faster than writing blogs about exploiting known issues.
Done.
* Claudinating...
Done?
This headline would not be out of place at the beginning of Terminator, foretelling Skynet going rogue.