Hacker News Reader

Comment by masterj | Hacker News Reader

Outside of the initial wave of security vulnerabilities and scrambling, it seems like the logical outcome of this over time is likely vastly more secure vm environments?

View on HN

I think the main problem with that is the main problem with a lot of security tools. In order to do useful work, you need to provide a lot of tools and permissions.

I.e. in theory the most secure might be a virtual machine with no network access. But then how do you access the LLM provider? Etc.

I suspect capability models are going to get more popular https://en.wikipedia.org/wiki/Capability-based_security

I'm doing some work in this space[1], it's a really deep and hairy problem. There's a lot that you can do at the OS level sand boxing of course, but you may want some areas of you code to have network access and for some areas that use external dependencies to not have that access. Tracking where systems have side effects ends up being a lot of book keeping and a lot of languages that implement an object-capability model require you to thread caps through all of your calls, which IMO is bad ergonomics and a place where bugs creep in as functions accrue caps. Or sometimes they have rather superficial cap systems like Hack, or are rather awkward like in Deno.

[1] https://march-lang.org/docs/capabilities

Why do you invent a new language for your work?

Why did you not embed your language into another one, with type system that is superset of what you need?

For example, there's capabilities expressed in Haskell: https://github.com/tweag/capability

Capabilities there are tracked at type level and are subject to type erasure, if possible.

There are a few reasons for this. I really can't seem to wrap my head around Haskell in the wild as written by real people. I also wanted to tie capabilities into the build system tooling so that the build tool could verify caps and launch executables into an OS sandbox. Bolting on capabilities doesn't offer the same ability to enforce them, for example the March package manager ForgePM will reject packages that falsify their cap manifest. Moreover I was reading some papers that inspired the language, and wanted to try it. Cap(X) is also erased b the type checker during compilation in March.

I'll blow my own trumpet and promote my own Haskell capabilities library, Bluefin: https://hackage.haskell.org/package/bluefin

Bluefin is used in production, and as far as I know capability is not.

That looks really cool. Can you narrow Bluefin.IO to reads/writes separately? One of the things I've worked on is the ability to allow code to read files, even specific files, but deny writes.

Yeah you can write a capability that encapsulates exactly whatever effects that you like!

Did this start as an effect system and then capabilities shook out naturally?

Yes! It started as an implementation of the effect system I always wanted: effects passed on the value level, rather than implicitly on the type level. Once I'd done that I realise that it was actually a capability system (and that was the better way of describing it, because more people already know what a "capability system" is).

That’s not surprising there’s a lot of mechanical overlap between the two. It’s a really interesting relationship.

Yeah, I think they're basically identical concepts. I'm writing an article on that topic right now.

I’d love to read it!

Here you go! It's a complete document, but I'll continue to edit it over the next few days before I announce it. I'd love to hear any thoughts you have.

https://h2.jaguarpaw.co.uk/posts/bluefin-capability-system/

Awesome, I'll take a look and shoot you an email if I have anything interesting to say about it.

Cool, my contact details are on the front page of that site.

That's a really interesting project! Being able to assert more, statically, about what our software is doing seems like a real growing need

Thank you, I really appreciate that. I'm really trying to make it easy to assert what code can do, then enforce it at compile time and optionally via the build tool at runtime via OS sandboxing integration. Supply chain attacks are also an area I'm exploring by having deps declare caps and then the build can scan and inspect binaries.

It's been a real education. I talked to one of the people behind Caja and learned a lot.

> But then how do you access the LLM provider? Etc.

You can expose an HTTP proxy over a vsock into the VM.

That sounds like more complex pieces of software that likely have security flaws and are easy to misconfigure.

This is the way.

If "LLM provider" is part of the conversation, then you have already have ceded the security question.

It's the security vs convenience trade off.

Invent new protocol just for LLM?

https://en.wikipedia.org/wiki/Model_Context_Protocol

That makes as much sense as saying that better gun technology results in body armor that can stop it. It might incentivize that, but in no way "results" in that; the fundamental technologys underpinning advancements in offense versus defense are fairly different.

Maybe, but I think this misstates how security vulnerabilities work. Vulnerabilities are logic flaws, and we have every reason to believe that there is a maximum number of such flaws in any given system that allow exploitation; and even that we can conceivably develop logic that excludes any flaws. Whereas weapons and armor are devices that deliver and deflect/absorb energy, and any increase in the power of one means we need a corresponding increase in the other.

Now maybe our understanding of logic systems is wrong, and it's just fundamentally impossible to develop programs that lack exploitable vulnerabilities -- that you can always "exploit with more energy". But there's no reason to believe the energy metaphor transfers to logic and intelligence.

Very poor analogy - information security has very few commonalities with ballistics (duh)

Except in this case, the gun is the one directly improving the body armor.

Only if you think penetrating holes in a paper vest and then patching those holes constitutes “improving”. If that worked Windows and Linux would be impenetrable fortresses with all the holes that keep getting punched in them.

Cyberdemolitions expertise is about as relevant to cybersecurity as gun making is to bulletproof vest making. Necessary for validation, but not very related to the fundamental engineering and technology.

We need better digital jailcells for our digital slaves basically.

Or if you see AI as more tool and less entity, better gunsafes for our guns.

They are more comparable to a computer worm than anything else. Very strange (and disrespectful imo) to make the jump to slavery. A gun can’t be used at all inside it’s safe so I’m not sure that makes sense. A better metaphor would be making sure gun ranges have backstops capable of stopping contemporary payloads and sufficient range controls to keep people from shooting at cars on the highway. Outside of that you need registration requirements and gun control to make sure you can mitigate and track down perpetrators of gun crimes off the range. If they’re to be used in active conflict you need laws of war to govern the use of lethal force. If you use them to hunt, you need a hunter’s safety card and a current tag.

A gun cannot fire unless someone loads and triggers it, an LLM cannot operate computer unless someone translates what it said to shell scripts. An LLM is super comparable to a gun, not a computer worm which is consistently dangerous.

Sure. I never said guns were a bad metaphor. I meant a worm was more comparable than an “entity” or “digital slave”. Agentic AI is putting the gun on a robot dog, taking the safety off and handing over fire control to an algorithm. If you extend it that far though guns are just any software. Or perhaps computers are guns and software programs are bullets, with LLMs manufacturing bullets from tokens.

maybe it's just information that really, REALLY wants to be free?

This assumes your malefactors don't do malicious engineering, injects, social-agent engineering, etc.

This same assumption is built around the singularity, the TAM of 30Trillion, etc. It's the idea that complexity will some how collapse upon itself in some bizarre borg like collective.

Entropy is still going to win.

Reply on news.ycombinator.com