If someone asked me what a DB is, I'd probably start the conversation with the exact description "a file-backed map, like a hashmap".
4,717 karma · joined September 16, 2022
Rapid7 -> Dropbox -> Grapl -> Datadog
Ultimately I have stuck to the informal verification of defining my expectations and ensuring that tests cover them.
This already describes how CEOs work though. CEOs aren't experts in every part of the company, they sign off on things and delegate, and manage communications etc. There's very little additional room to delegate because the role already is extremely delegation heavy.
I'd have loved for an AI to assist with a few things as a CEO but we would have probably just shipped faster and it would have created even more pressure for board communication, customer calls, design partnerships, legal discussions, compliance, etc.
The idea that AI is coming after leadership is truly laughable. The idea that it's coming after some engineering jobs is a bit more likely to me, but I think many will actually be more valuable.
Yes, it's an amazing feat that has cost billions and led to major features like seccomp v2, ptrace sandboxing, etc. Do you know the history of browser sandboxing? It's pretty complex, a major technical feat.
> Don't sit there and tell me that bwrap, sandboxing, whatever is some kind of recent innovation with uneven distro support.
I can tell you that unprivileged sandboxing is not only new, it's ongoing and nascent work and not commonplace at all.
> And yes, bwrap needs root or namespaces. So what? So does sudo. Setuid binaries are as old as time.
These are massive footguns and issues for deploying code... You don't get that? Again, implementations have tradeoffs.
> You know what's also been around and doesn't require root? Landlock.
lol it is SO NEW what the fuck dude?
> Have you heard of it? Has anyone?
Yes, I use it!
> "Oh, the technology works, but it need setup! Oh, my distro doesn't enable it! Oh, it doesn't work out of the box with my code!"
> This is a social problem.
Yeah these sound like social problems if you have literally no idea what you're talking about lmfao. IT's BOTH.
Your ignorance is blatant and your position is dead in the water.
Then obviously we have to discuss the implementation!
> Or just invoking bwrap. Or sandbox-execute. There's no performance overhead. Complexity is minimal. Just read current Codex source code. It's not so bad. People have been making these sandbox tools for AI agents for years. Just need to apply sandboxing to all domains.
Tools like bwrap literally could not have existed until very recently without also requiring suid/ privileges, and even today unprivileged user namespaces are not universally enabled. This is why the implementation matters. No, unprivileged sandboxing has not been around for years, especially not x-plat. bwrap is a perfect example of what I'm talking about, great reference - it either requires root or it requires unprivileged user namespaces and it's not x-plat. Great.
> You want some kind of silver bullet that makes code "safe" without anyone having to change anything? To prompt for nothing? To use no IPC portal? That's not happening.
I have no idea why you think I've said this, I'm pointing out that the implementation and technology matters deeply.
> The people saying we need a new language or something are half right. We don't need a new language. A pure library solution is fine! But people do need to do work to make an ecosystem based on capabilities and least privilege to work. And we need to tell people who refuse to do this work to go to hell.
Again, you can't say "this technology has existed for years! You can do it in a library!" and then say "but they don't!". You misunderstand the complexity and assert that it's merely a matter of will. It is both.
Sandboxing almost always involves having a dedicated, privileged service, due to how operating systems have designed things. It requires platform specific code.
This is a technical problem and a social problem, there's zero reason to believe it's just one.
This is why I'm building this language. It's capabilities based. Build scripts can't just do whatever the hell they want to, everything is auditable, and it layers its sandboxing.
It's for fun and anyone looking should understand that this is AI driven building with human driven design, but the goal is to demonstrate.
> The thing is when AI goes to hack 'all the things' it only needs to pick the weakest link in the stack and your house of cards falls down
Yes, but you can... mitigate the risks? I've explained this.
> The other flaw in your plan is that people make mistakes, a lot of them, all time
Yes, you mitigate the risks. That's why you layer things.
> I spend $x on security won't save you
No one is saying this.
> AI already hacked Hugging Face with brand new zero days like it was nothing.
No, it cost OpenAI money, and those zero days are unsurprising and probably are like ~O(10K) at human level.
> The real bad actors - malicious AI will find the one flaw, on that one server, in the corner you never thought about and turn your network inside out with it faster than it takes you to have the standup meeting about the weird anomaly detected while you all were at lunch.
Science fiction and not supported. The vulnerabilities found by AI are not surprising in the slightest.
I've made my point abundantly clear.
Rereading this, it's an overstatement. It doesn't "virtually eliminate" it. It drives the cost up by like 3 orders of magnitude and it pairs well with other mitigations. gVisor escapes can and will happen, I highly recommend you do more than just stuff something into gvisor - additional seccomp can go a long way, and apparmor/selinux pair incredibly well.
This is obvious. AI doesn't drive the cost to zero and exploitation has always been about cost. Tokens cost money, not everyone has $10M to burn on chaining bespoke 0days.
Consider that if the cost of exploitation was truly 0, then the cost of perfect software would be 0 because you could exhaust an infinite search space of vulnerabilities for 0 cost. Your conclusion could never follow from your premise.
Yeah but that's a business decision. I work on security at a company that does sandboxing and when the company decided to build an AI harness I was brought in as one of the earliest engineers on the product. We do almost everything on that list and we're a fraction of the size of OpenAI. And it wasn't particularly hard, and we have harder requirements imo (because we solve more general problems vs "run a very specific agent with a very specific task and very specific access").
> And you’re also not fully considering the granularity problem, eg there are a lot of sandboxing tools out there but they’re usually quite coarse in the dials and levers they offer, so the only way you can still make the workload do what it needs to do is tune them relatively permissive.
Very little software is incompatible with running in gvisor, for example. Most people can just overwrite `runc` with `runsc` and things will "just work".
Running an artifact repository in isolation isn't particularly novel or complex either. You can virtually just eliminate SSRF vulnerabilities with a host based firewall or AWS Security Group etc, like the whole problem goes away by just saying "this box can only talk to that box".
Tools like Smokescreen exist, they work great, they're super easy to deploy. I bet OpenAI could do it, I bet they could run 500k tokens just fuzzing and eval'ing it for 0days for like 48 hours before they actually deploy it too.
OpenAI as a business chose to not bring people who know these things in, or didn't empower them, or didn't prioritize it organizationally. I'm not a genius for saying "use gvisor, set up a firewall, isolate resources" - I'm quite sure there are people over there who would get it done in a weekend. But they didn't, and that's notable.
This is incorrect. It's actually the whole point. Imagine you're an attacker in a gvisor container with a Firecracker hypervisor around you, and a proxy on the host holds a signing secret that gets exposed through the VM virtual device.
Getting access to that secret is not one crack. You need to escalate out of gvisor. That likely gets you control over the Sentry process - let's ignore its sandboxing and just say "you're an unprivileged user".
Any viable attack on Firecracker requires either KVM / hardware exploits (>$1M but definitely real) or has to start at the kernel. Okay, that's about 10-50k to get a kernel LPE, maybe 5K in tokens these days.
So you're in the kernel in the guest of the VM. Time to expoit firecracker lol. It's... never been done. There are like two promising CVEs ever and they're not actually exploitable, no one has done it. Okay, so like, hand waving, let's say it's about $1M to exploit firecracker.
Great, you're unprivileged on the guest. We'll just kind of ignore the additional sandboxing that Firecracker does.
NOW you can try to attack the proxy by scraping its memory or whatever.
This is literally millions of dollars for standard infrastructure hardening and you could go so much further. You can trivially make kernel exploitaton 10x harder, you can make gvisor escapes much much harder, you can move the proxy signing into a TPM (depending on requirements but whatever), you can move the proxy to another computer altogether, you could fuzz these systems for days or run agents against them or whatever.
But one thing is certain - it is never "one crack".
This is absolutely not true. It's a matter of cost. Exploitation can cost on the order of 10K, 100K, 1M, 10M, etc. A straightforward one would be something like "MD5 collisions are on the order of $100K-1M" (a while ago, at least) so if you used MD5 you knew that it costs about that much to bypass the control. Moving to SHA1 pushes you massively out of that space, even if that algorithm has flaws.
I'm sure that Firecracker has vulnerabilities. Cost of exploitation is likely >100K, likely >1M. gVisor is likely on the same order of magnitude and these two technologies stack because they address the same surface and can be used in conjunction.
Software absolutely doesn't have to be perfect, it just has to be costly to attack and it's hilariously easy to drive costs way way way up.
Yeah, basically. I mean I'm handwaving but yes, some combination of those would have made the attack way too expensive.
I could go on and on. A tiny bit of forethought and effort pays off massively.
I personally feel that we already have the technology required to contain AI, it's just poorly leveraged. Tools like gvisor have existed for ages but are rarely deployed, Firecracker has existed for ages but is rarely deployed, seccomp has existed for ages but is rarely deployed, memory safe languages without decades of serialization vulns have existed, capability-safe libraries have existed, iframe sandboxing, trusted types, content security policy, network ACLs, isolating proxies, fuzzers, formal verification, refinement types, etc.
It's crazy just how safe software can be if you put the effort in. With AI I think we're just seeing how little anyone has bothered to leverage this tech.
OpenAI put shared JFrogy infrastructure in front of their sandbox. I mean, really? Whipping up a hardened artifact infra project with AI is trivial these days and it could have had 1% of the attack surface, been totally network isolated, totally infra isolated, fuzzed, sandboxed, etc. Why didn't they? Stuff like this feels inexcusable for a company with effectively unlimited tokens. I've literally done this with a "pro" subscription.
Show me an AI that breaks out of gvisor wrapped in Firecracker with an credential-injecting proxy and real network isolation. We already know that Mythos couldn't do it - the vulnerability it found in Firecracker required incredible effort and positioning just to not be exploitable. I'm not saying there are zero vulns in it, but the cost is insane.
It's INSANE to me that OpenAI has to say "we now use proper sandboxing". To be frank, it's a bit disgusting to me. I've recently built an AI sandbox and gvisor was just the start of that conversation. If I were OpenAI training hostile models I'd probably start with gvisor, harden further, and potentially consider the entire piece of hardware compromised - they can afford this, they could reflash firmware after evals etc.
I have no clue what you're talking about and I doubt that it's relevant.
The claim was made that this is "simple". It doesn't feel simple to me. I'm asking about the criteria used and how this fits into that to justify the "simple" claim.
The implementation details of this construct are implementation details of the language.
Of course, they don't have to learn anything other than "use `go vet`, it's what enforces this" to use it. But that's true of almost anything?