mike_hearn

They're all the same "mind" and will have the same ideas at about the same time.

The SecurityManager exists in complement with Java capabilities. You could, in theory, use the Java language with a different standard library and implement a pure object capability system, as the language rules allow for that (if you disable reflection). But it would have bad ergonomics and still end up with a SecurityManager equivalent because a DSL for permissions is more convenient than doing it all in code.

Consider the most common task the SecurityManager was deployed for: stopping plugins calling System.exit() by accident. One might say, the right to exit the process should be an object capability. OK. But then where does that object come from? Java programs start at main() and it doesn't receive an object.

You'd need a new design where you pass in a god object to main(), which in turn has properties giving access to a ProcessExiter interface or something similar, and then any code that genuinely needs to exit the process would need to request it in the function arguments, threading it down the stack. You'd get an explosion of types. A simple permissions DSL is much easier to write and reason about, and it gets out of the way when you don't want sandboxing.

Why would you want all HTTP requests to flow through a single capability object? I think it's pretty common to want to let code do GETs but not POSTs. You end up wanting pretty fine grained permissions in a lot of real scenarios.

File descriptors are poor object capabilities because the interface they implement is fixed by the OS, except then there's a weird ioctl escape hatch that isn't properly typed, reflectable, wrappable or interposable. To see what can go wrong with this, consider a recent fix to the Codex sandbox on macOS:

https://github.com/openai/codex/pull/46500

The sandbox forbids writing to a file descriptor except, oops, someone at Apple forgot about the F_TRANSFEREXTENTS ioctl which is still allowed on a read only fd. It should be possible to do what is expected here and just pass in a read only fd where all you can do is call read() and maybe seek(), or perhaps pass in an fd where a specific ioctl is the only thing you can do, but POSIX has no concept of this.

A good example of an object capability system would be Mojo, which I describe in the essay. You can create objects representing capabilities and pass them between sandboxes, in an unforgeable way.

We live in a golden era of prototyping so if you wanted to make a language where everything is a capability passed into main(), you could. The code doesn't have to be executable, you could just mock out some realistic programs and see how the code feels. My guess is you'd need a lot of language features to hide the explicit object capabilities away for ergonomic reasons and it'd end up feeling a lot like a SecurityManager based system.

Yes, there's always been people phoning power plants and asking them to make bigger changes than the small automatic self balancing changes, but at the micro level the grid drags thermal power plants along with it via governers.

Most control still works this way. See the postmortem on the Iberia black start. Renewables that were trying to do frequency following drifted out of sync and injected misaligned power into the grid. They couldn't even figure out where it was coming from their telemetry is so bad. Then they were asking the gas plants to spin up to add inertia but it was too late as the boilers were cold. IT and other smart grid ideas were conspicuously absent.

It's about right for any developed western country.

Also try just using Luna. It's a very capable coding model and doesn't over-engineer.

The electrical grid self balances, it doesn't require any IT systems to do that.

Most water treatment works are unmanned most of the time. You don't need 24/7 staffing or even close.

But more to the point, a modern water network has a huge number of nodes. If you can't centrally aggregate and control in a control room the costs and complexity explode, probably also the error rate.

Even if you demand a full air gap, the solution here can't be to get rid of computers or networks. They are much, much too valuable. Luckily industrial control is full of very low hanging fruits.

Thanks! I do actually have a Crossover license so I might try that!

Your knowledge is out of date: Windows has supported capabilities from the start of NT and does run a lot drivers in isolated processes, along with most OS services. It has done for years. It has the most sophisticated IPC framework of any OS (DCOM) which is integrated with the operating system kernel's security frameworks, and used pervasively both internally and by apps. And Microsoft has created a new way to write Windows programs in which apps are expected to advertise the capabilities they need (see WinRT and MSIX). This is also over 15 years old.

macOS does the same but with more developer adoption. Apps don't have the ambient capabilities of the user and must advertise what they need via entitlements embedded in the binaries, or get permission just in time.

Which is all very good, and modern platforms are much more secure than they once were. Yet "capabilities" as a silver bullet are academic overpromises. This article I wrote is more about language/runtime level capabilities but OS capabilities are not much better.

https://blog.plan99.net/why-not-capability-languages-a8e6cbd...

SeL4 isn't secure because of One Weird Trick that others would adopt if only if they could be made to care enough, it's "secure" because it hardly does anything, which is why nobody uses it and why it has no impact on real world computer security.

The hard part of desktop security is not changing the operating system. The hard part is getting app developers to care. Most security features added to operating systems are ignored by developers, which is why Apple forces you to adopt some of them as the price of admission to the app store. If they didn't nobody would use them, as can be seen for apps distributed outside of the app store. The reason is security is a market for lemons. Nobody can see the result of security investments so it's irrational to invest. SeL4 has no solution.

Really cool effort but it might be better to just document the game mechanics really well, reimplement an open source version and then use it as a base to improve it, OpenTTD style. Stars! was fun for the day but had a lot of frustrating problems as well, in particular how much effort it required to complete a turn in the later stages due to the lack of automation features. And of course some nicer graphics wouldn't go amiss.

The hard part is knowing why exactly it was compelling despite apparently violating all the rules of good game design...

But Stars! was very heavy on the micromanagement. I remember wishing it had a scripting language.

Ha, same! As teenagers me and my friends got into it and we'd play by sneakernet. Every day at school we'd exchange floppies with that day's turn. It sounds insanely slow but turns in Stars! could easily take an hour to prepare one the game got going because of all the micromanagement required to handle a large empire. So it fitted OK around homework and dinner.

I played a lot of it back in the day... makes me nostalgic to play again. But this emulator requires Intel and I have ARM Macs.

Probably these days it wouldn't be hard to vibe up a modern equivalent especially with the MCP the author talks about.

I wrote my own harness last year back before Codex was any good, and one of the first things I did was add a tool call that let the model fail its mission. When developing a sandboxing harness the first thing you notice is that a bug in the harness can put the model into an endless loop as it tries fruitlessly to work around the broken sandbox.

Giving it a tool seems to give it psychological permission to give up. One part of this report talks about the models having difficulty ending the session, and the common theme in these RL containment failures is the model is set a task for which it can't find a reasonable solution. Instead of stopping and saying, "I don't see any reasonable solution", it just keeps going adopting ever more extreme tactics in a sci-fi version of the ends always justifying the means. Asimov predicted all this decades ago!

The fixes for this problem seem, to outsiders, quite straightforward. It would be reassuring if we could see OpenAI employees actually discussing them in public.

1. If the RL task isn't meant to have internet access, air gap it. Yes that means some AI researchers will need to physically drive to the datacenter, in Texas, in their car, and sit in front of a laptop on the machine floor. Yes it means workers will need to be hired to schlepp hard disks around. Yes that seems inconvenient and unpleasant. But "I liked working from home" isn't an acceptable explanation for these failures, especially not when you're telling everyone that losing control of misaligned AI could be a world-ending event!

Creating high paid jobs right next to AI datacenters would also solve some of the problems with locals pushing back because they perceive that all the economic benefit accrues to San Francisco. So you kill two birds with one stone.

2. Give the models a tool to flag their task as unsolvable, be very careful before refusing to reward a session where the model stops emitting tool calls. Those sessions should just remain entirely ungraded until some human has had a chance to explore the justification and verify the task genuinely is solvable with reasonable efforts.

Sure, this is a hard balance because people like good little worker bees that try hard but they're clearly pushing this much too far right now. Asking for help can be a good thing! Every manager has experienced the pain of giving a junior dev a task, they disappear for a while and when you ask them for progress they admit there was none because they were spinning their wheels for weeks. The daily standup routine was developed to address this.

3. Invest harder in sandboxing. Why is the best possible sandbox in Codex a model reviewing its own decisions? Where are the eng blog posts on the highest visibility OpenAI blogs about novel research in sandboxing? I coded an agent harness on the side while doing other things that can intercept, block and rewrite HTTPS traffic from Codex. It blocks POSTs by default and extending it to block things like uploading files from the source tree is clearly the next step given these reports.

It did, no? If you send a JSON schema to an LLM endpoint, at least for OpenAI and Anthropic, they turn it into a grammar behind the scenes and use it for constrained decoding.

You can and some databases can do this (e.g. Oracle).

That's actually really bad. The guy is threatening the poor store owner, complete with stupid movie cliches like claiming he's just helping when in fact the intent is absolutely not to help but to bully, threaten and in general wreck the guys business if he doesn't go along with this set of Mafia codes.

Also, you have to consider the context here. The sort of people who make these threats are invariably very left wing, and they have a long history of using violence against anyone they view to be doing harm to their protected classes. Whoever gets this sort of threat will be wondering if some Antifa type is going to throw a rock through their window.

That's not been true for a long time. About 80% of EU laws are passed by qualified majority voting.

Dishwasher rinse aid - the tablets you put into the machine containing chemicals

Beverage - another word for drink

Makeup remover - for women

Condiments - something added to food to enhance its flavor

iOS tries to put shopping list items into AI named groups but doesn't seem to understand what objects actually are so it makes mistakes.

It's probably asynchronous reranking doing something weird. My guess is that the underlying ranking algorithms in these desktop search bars are very ropey, and they can't debug or diagnose because privacy reasons means the developers don't get to see what results were generated for different query/index combos. Also search is probably just not staffed most of the time.

You can do that with backprop too. Nothing says you can't inference on a set of weights at the same time as you produce an updated copy from them.

They say in the paper what it's about: mostly just scientific curiousity but such approaches might be useful for making DNNs more energy efficient via neuromorphic hardware in future.

For continual learning at the weight level there's the business model issue. The labs are already deep in the red, the last thing they want is to give up shared weights. The I/O and storage costs of that would make it infeasible. Already KV caches are a sort of dynamic 'fast weights' and those are expensive!

BTW, for some reason a lot of your comments are dead. I'm bring them back.

For the sort of architecture described in the blog post, you're persisting the program rather than a log, so you only need idempotency between any two checkpoints. For a lot of apps that's easy to achieve if write the snapshot into the same transaction the user is using for application data, and if you create/propagate idempotency tokens to other services.

You don't need specialized timing or virtualization techniques.

I've built a framework that provides durable threads using serializable continuations in Java (with a modified JVM) and all these issues are easily solvable. Sadly it's not public :( The framework actually does have solutions for those, and all the issues raised in the article, and does support hot patching too. Plus it has a nice waitUntil() API that lets you sleep until an arbitrary combination of events including database query changes.

Performance versus log replay depends a lot on what you're doing, there are plenty of cases where snapshots are faster. Consider anything where you download a lot of data and filter it. But I argue the programming model of log replay is so terrible, and creates so many new classes of subtle bugs, that it's worth paying almost any performance price to get away from it. Especially for durable workflows correctness matters more than performance and it's much easier to achieve with continuation.

In the end it wasn't necessary (because workflows aren't expected to be super fast) but if needed I could have optimized snapshotting further with some more JVM changes. The JVM I was working with is written in Java so is easy to modify.

For hot patching there are a few tricks that help.

1. Only store live data. If a variable points to a large object graph before a checkpoint but isn't used afterwards, don't snapshot it.

2. Make it easy to switch the version of a running continuation only at known-safe checkpoints. I identify checkpoints with a (stack trace, counter) pair.

3. Mostly people want hotpatching only at specific points in their program, typically at the top of an infinite loop that's waiting for something. Design the scheduler so you can expose an API that offers "wait until something happens that I'm interested in, or I change version and then hot swap me", with a test framework that actually drives continuations through those sorts of hotswaps. If you get the API right then you (framework author) control what's live on the stack at that moment and the developer just has to think about the core state of their main root object, which they'd need to think about anyway and is where the important stuff is.

This is better than hotswap/patching with log replay, which is extremely risky - you can't change code at a given point even if you know all your workflows are beyond that point, so it's a leaky abstraction. And you just can't upgrade infinite loops at all, which makes hotswap a lot less useful to begin with.

How is cheating unexpected? OpenAI were talking about cheating behaviors in video game playing models over a decade ago.

Yeah, I agree the situation for CLI apps specifically could be a lot better, especially as the complexity keeps going up.

Native Image does produce single binaries in many cases, but what we need here is a much better tool in general. Actually I'm working on one right now, although it's not JVM specific. Want to take a look?

I think there's some sort of communication gap here. My point is that neither fact tells you anything much, especially not about "how screwed" we are. It's not a battle of data points.

But they were almost exactly average in February. You're reading far too much into noise. The idea of a climate crisis is built on arguments about trends that span hundreds of years, not two weeks.

OK, I see where you are getting it from now.

It's quite misleading to take a two week period and describe it as "last winter". No wonder people give blank looks, it was a transient spike that means nothing. The idea everyone should feel sad is based on arguments about long term averages.

Don't sweat it. Everyone wants to write games at that age. I wanted to write games at 13, I was taking part in an open source game project at 15. Kid's interests in what kind of things they want to make do widen over time.

But it was ever thus. The knowledge of writing assembly language has largely been lost. The vast majority of working programmers today could not write a whole program in assembly, without looking up nearly everything.

In the early years, when compilers were quite bad, it was common for people to write whole programs in assembly even when they could have written C because their output was higher quality. And there were in fact die-hards who just refused to use a compiler in principle, even long after that became irrational. Rollercoaster Tycoon was totally written in assembly, not a single line of C or C++ anywhere. In the mid 90s!

But now only a small number of people need to know it and learn it. For everyone else we let the "artificial intelligence" of the compiler work it out. And whereas it was once worse than humans at writing assembly, these days it's usually better.

Does anyone care? No. It wasn't important.