jrtc27

> Unfortunately, the time between free and revocation introduces a short-but-not-zero window for UAF bugs/attacks. This time gap is even explicitly acknowledged in the Reloaded paper!

Yes, revocation is batched and asynchronous. This does mean that capabilities remain valid beyond the language-level lifetime of the allocation. However, that does not mean that, within that window, we have not dealt with any UAF attacks. The vast majority of UAF attacks do not care about the fact that the memory has been freed, but rather that the memory has since been repurposed for something else (whether the allocator's own internal metadata or some other new allocation). Cornucopia (both versions) ensures that this does not happen until the next revocation pass; that is, it "quarantines" the memory. Effectively, when you call free, it's "as if" the free were deferred until revocation time. Therefore, if your capability is still valid, that memory is still only in use by you, and so the vast majority of attacks no longer work. This protects you against UAF in a similar way to how making free a no-op protects against most attacks. This is not all attacks, very occasionally the bug is a result of something like undefined behaviour that follows, but I don't know if we've found even one real-world instance of a UAF that this approach isn't going to catch. I'm sure they exist, but the nuance is crucial here to be able to reason about the security of various models.

But yes, MTE+CHERI are complementary in this regard. We have drafted ideas for using MTE with CHERI, which would (a) let you immediately prevent access (noting though that the capability would remain valid for a while, still) (b) let you recycle memory with different MTE colours before needing to quarantine the memory (hoping that, by the time you run out of colours for that memory region, a revocation pass has reclaimed some of them). That is, in theory it both gives stronger protection and better performance. I say in theory because this is just a sketch of ideas, nobody has yet explored that research.

I also note that MTE does not fix the undefined behaviour problem; it will only trap when it sees a memory access, but vulnerabilities introduced due to compilers exploiting undefined behaviour for optimisation purposes may not perform a memory access with the pointer before it's too late.

You may wish to read what the current pure-capability CHERI Linux user ABI specifies for mremap(), because we (primarily Arm, in conjunction with us) have thought about this, and the conclusion is not "the existence of mremap() makes CHERI undeployable". See https://git.morello-project.org/morello/kernel/linux/-/wikis...

> Does that study include enabling intra object overflow protection, or not? > > When I say that this optional feature would force you to change a lot more code I’m comparing CHERI without intra object overflow protection to CHERI with intra object object overflow protection.

Sorry, I misinterpreted what you were saying. No, that's not with subobject bounds. If you want that then yes there is more incompatibility, because C does not have a good subobject memory model. That's not really because there's anything wrong with CHERI, it's just because the language itself is at odds in places with doing that kind of enforcement with any technology. But, if you're willing to incur that additional friction (as we do for our pure-capability kernel in CheriBSD), you can enable it, and it can protect against additional vulnerabilities that other security technologies fundamentally cannot. We even provide a sliding scale of subobject bounds enforcement, where each of the three levels restricts bounds in more cases at the expense of compatibility. The architecture gives you the flexibility to decide what software model you want to enforce with it.

> Finally, 6 million lines of code is not that impressive.

We have far more than that ported, that was just one case study done in a few months by one developer. FreeBSD alone is, by my very rough estimation cloc that excludes LLVM, about 14 million lines of C and C++ (yes, I'm not distinguishing architecture-specific code and all kinds of other considerations, but it's close enough and gives an order of magnitude for the purposes of this conversation), and we have FreeBSD ported. Not to mention our work on, say, Chromium and V8 (Chromium being another set of 10s of millions of lines of code, again tractable with the engineering effort of just a few members of our research group).

> Real OSes are measured in billions

Citation needed. The Linux kernel is only a bit over 40 million lines of code these days. Real systems may well approach the billions of lines of code running once you factor in all the libraries, daemons and applications running on top of it, but that is not all low-level OS code that needs the kind of porting an OS or runtime does. Even if it were a billion lines of code, though, extrapolating at 0.026% that would be 260 kLoC changed, which isn't that scary a number.

Even V8, which is about the worse case you could possibly have (highly-stylised code written in a way that uses types in CHERI-unfriendly ways; a language runtime full of pointers; many (about 6?) different highly-optimised just-in-time compilers that embed deep knowledge of the ISAs and ABIs they are targeting and like to play games with pointers in the name of performance) we see (last I checked) ~0.8% LoC changed, or about 16k out of 2 million. The porting cost is real, but the numbers have never suggested to us it's at all intractable for industry.

Yeah you need a compiler, linker and OS. That's true of any security technology. CHERI may be more significant in that regard because it's a bigger rethink than just stuffing some extra metadata into the existing types, but it's not at all intractable. We, a research group, maintain CheriBSD, a "full-fat" port of FreeBSD to CHERI (Morello and CHERI-RISC-V), so to a big tech organisation it's a small investment. The cost to tech companies is not making it work, it's often much more boring business factors.

No. The capability itself lives in normal memory intermingling with data just like any other pointer. There is no "capabilities memory", it is just memory.

In MTE, you have the N-bit (typically 4) per-granule (typically 16 byte) "colour"/tag that is logically part of the memory but the exact storage details are abstracted by the implementation. In CHERI, you have the 1-bit capability tag that is logically part of the memory but the exact storage details are abstracted by the implementation. If you understand how MTE is able to store the colours to identify the different allocations in memory (the memory used for the allocations, not the pointers to the allocations) then you understand how CHERI stores the tags for its capabilities, because they are the same basic idea. The difference comes in how they're used: in MTE, they identify the allocation, which means you "paint" the whole allocation with the given "colour" at allocation time (malloc, new, alloca / stack variables, load time for globals), but in CHERI, they identify valid capabilities, and so only get set when you write a valid capability to that memory location (atomically and automatically). This leads to very different access patterns and densities (e.g. MTE must tag all data regardless of its type, whereas CHERI only tags pointers, meaning large chunks of plain data have large chunks of zero tag bits, so how you optimise your microarchitecture changes).

Perhaps you're getting confused with details about the "tag table + cache" implementation for how tags can be stored in commodity DRAM? For CHERI you really want 129-bit word (or some multiple thereof) memory, but commodity DRAM doesn't give you that. So as part of the memory controller (or just in front of it) you can put a "tag controller" which hides a small (< 1%) fraction of the memory and uses it to store the tags for the rest of the memory, with various caching tricks to make it go fast. But that is just the tag, and that is an implementation detail for how to pretend that your memory can tag data. You could equally have an implementation that uses wider DRAM (e.g. in the case of DRAM with ECC bits to spare). Both schemes have been implemented. But importantly memory is just 128+1-bit; the same 128 bits always store the data, whether it's some combination of integers and floats, or the raw bytes of a capability. In the former case, the 129th tag bit will be kept as 0, and in the latter case it will be kept as whatever the capability's tag is (hopefully 1).

Where studies suggest "a lot" is sub-0.1%. For example, https://www.capabilitieslimited.co.uk/_files/ugd/f4d681_e0f2... was a study into porting 6 million lines of C and C++ to run a KDE+X11 desktop stack on CHERI, and saw 0.026% LoC change, or ~1.5k LoC out of ~6 million LoC, all done in just 3 months by one person. That's even an overestimate, because it includes many changes to build systems just to be able to cross-compile the projects. It's not nothing, but it's the kind of thing where a single engineer can feasibly port large bodies of code. Yes, certain systems code will be worse (like JITs), but the vast majority of cases are not that, and even those are still feasible (e.g. we have people working with Chromium and V8).

Can you elaborate on what you perceive as broken?

To reiterate what I've said elsewhere, CHERI does not need a whole parallel memory architecture, there is just one that gets a slight extension over a non-CHERI/MTE system to include tags. But that is the same story as MTE, which also needs to propagate the tags in the memory system (and in fact, more tags, since we just need one bit per 16 bytes, whereas MTE needs 4 bits per 16 bytes in the common scheme).

Cambridge and Arm have made a joint statement that nothing that is essential to the deployment of CHERI ("capability essential IP") is being patented by them: https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-953.pdf. As with any patent issues, you should consult your legal team and not take anyone else's word for it, because patent law is a minefield and who knows what patents may be out there lurking that nobody realises happens to cover some aspect of CHERI, or design choices in an implementation of it, as with any processor technology, but we are not out to patent it. We believe that the right thing to do is to make the technology open in order to allow it to be widely used for the good of the field.

> has a parallel capability stack

There is one stack, the normal program stack that's normal main memory.

> capability pointers

If you use pure-capability CHERI C/C++ then there is only one type of pointer to manage; they just are implemented as capabilities rather than integers. They're also just extensions of the existing integer registers; much as 64-bit systems extend 32-bit registers, CHERI capability registers extend the integer registers.

> requires microarchitectural support for a tag storage memory

Also true of MTE?

> your memory operations also need to provide a pointer to a capability object stored in the capability store

There is no "capability object stored in the capability store". The capability is just a thing that lives in main memory that you provide as your register operand to the memory instruction. Instead of `ldr x0, [x1]` to load from the address `x1` into `x0`, you do `ldr x0, [c1]` to load from the capability `c1`. But `c1` has all of the capability; there is no indirection. It sounds like you are thinking of classical capability systems that did have that kind of indirection, but an explicit design goal of CHERI is to not do that in order to be much more aligned with contemporary microarchitecture.

> The capability store is literally a separate bus and memory that isn't accessible by programs,

As above, there is no separate bus, and capabilities are not in separate memory. Everything lives in main memory and is accessed using the same bus. The only difference is there are now capability tags being stored alongside that data, with different schemes possible (wider SRAM, DRAM ECC bits, carving out a bit of main memory so the memory controller can store tags there and pretend to the rest of the system that memory itself stores tags). To anything interacting with the memory subsystem, there is one bus, and the tags flow with the data on it.

That's not true. Capabilities are in main memory as much as any other data. The tags are in separate memory (whether a wider SRAM, DRAM ECC bits, or a separate table off on the side in a fraction of memory that's managed by the memory controller; all three schemes have been implemented and have trade-offs). But this is also true of MTE; you do not want those tags in normal software-visible main memory either, they need to be protected.

We actually have ideas for how to combine the two; see section C.5 of https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-987.pdf

64-bit size_t is the same on Darwin as GNU/Linux (unsigned long), but uint64_t is not (Darwin defines it as unsigned long long, not unsigned long). Perhaps this is what you remember?

Stop-the-world is only brief, to scan the register file of each thread and any other capabilities held in the kernel (asynchronous I/O, signal handlers, etc). Once done the entire memory sweep can be done concurrent with execution.

As for porting, it very much depends what you're doing. Operating systems and language runtimes, especially those with JITs, have intimate knowledge of the architecture and like to play cute tricks with pointers, so those are disproportionately involved to port. General user code requires very little, if any, porting. In one study, a basic KDE+X11 desktop stack was ported to CHERI, seeing 0.026% LoC changes across 6 million LoC, or 1584 lines. It's non-zero, and of course there is a lot of code out there so even a tiny fraction of it isn't insignificant, but it is very small as these things go.

CHERI can provide heap temporal safety to protect against use-after-free (really, use-after-reallocation; use-after-free is harmless until the point at which the memory is being used for something else, and deferring lets you batch revocation sweeps), it's just not on by default yet as it's a bit too experimental, but we're working to stabilise it more for our next CheriBSD release.

The hardware must, and does, ensure that the metadata (both addressable - bounds, permissions, etc - and non-addressable - the tag) is kept atomic with the address portion of the capability, as otherwise you would be able to forge capabilities via such races. That is, you will never see a torn capability write, and the tag is updated atomically with every write, capability or not. This is easy to do since capabilities are always within a single cache line.

The SoC IP used in ASUS's Tinker V deliberately violates the virtual memory spec in a way that affects userspace, rendering certain virtual addresses unusable. This region overlaps with the default base address of position-dependent executables, so those cannot and will not run on it unless rebuilt with a different explicit base address.

> Doing this at the actual hardware level (as you're implying with registers) doesn't really work - there's too much variation in how languages work, how languages work changes over time, and it's also extremely expensive - in cpu time, memory overhead, and silicon area. Organizations have tried it in the past and I don't recall any actually being successful, except as examples of why "CISC is bad" :D

That is precisely what our research does, and Arm have built an experimental prototype, Morello, that is real silicon (4 cores, 2.5 GHz, based on the Neoverse N1 seen in various high-end Arm server offerings, including AWS's Graviton2). Our belief is that it does work and that it isn't extremely expensive.

They don't ship with spacers?

Oh I know. Hardly inspires confidence; all of those examples are clear bugs declared features because of poor programming practices.

> The if directive is tricky to use, especially in location{} blocks. It often doesn’t do what you expect and can even cause segfaults.

That's not user error. That's lazy programming and a bug. If your code segfaults because of user input you weren't expecting, that's on you as a programmer.

CHERI is not an actor system. It is a capability system aimed at memory protection. It can be used, like any other architecture, as a basis upon which to build an actor-based framework/system, but it is no more of an actor system than, say, x86. The concepts are deeply rooted in the capability system literature.

Your research on actor-based programming models has nothing to do with C/C++ spatial and temporal memory safety.

Yes it's non-zero, though https://www.capabilitieslimited.co.uk/pdfs/20210917-capltd-c... is a recent exploration of what it takes to port X11 and KDE to CHERI. Of the around 6 million lines of C and C++ code involved, only about 0.026% needed to be touched, or just under 1.6k. That number will of course vary significantly between the type of code; boring applications code generally doesn't need changes (e.g. htop and sudo built and ran out of the box for me recently, as examples), but language runtimes will need significant changes. Pages 21, 22, 26 and 27 of that report have the per-component breakdown of that number.

Yes, though "software" is rather broad; where exactly the bounds setting happens is important as if you get it wrong it allows malicious software to not set bounds and be able to access memory outside of its allocations. Pushing it to the same place the actual allocation happens or, in the case of referencing global variables, the same place the loading and relocating happens, ensures that the only thing malicious software can do by not setting bounds is make itself insecure.

You should indeed run sanitisers during testing and catch most of the issues; we encourage this! What CHERI provides is twofold:

1. Memory safety issues not found in testing do not lurk as exploitable vulnerabilities; testing is never perfect, often far from it when it comes to edge/unexpected cases where vulnerabilities lurk (though fuzzing can help somewhat)

2. Sandboxing still needs some kind of isolation primitive, which CHERI can provide in place of the heavyweight MMU-based techniques that exist today

Plus let's not kid ourselves that all software is being tested with sanitisers. The vast majority of software running on your system probably is not.

Then great, they push for the adoption of the technology and the world's computers become more secure. Arm's a big corporation and they're obviously pretty involved, and both Microsoft and Google are invested in the project.

They're relevant because they're technologies relating to memory safety and provide some level of additional protection. However, they rely on secrets and are in general only probabilistic, so they don't deterministically mitigate all memory safety issues (you can deterministically mitigate some with clever allocations of memory "colours", but not all). CHERI and MTE-like schemes also both rely on the use of tagged memory, but in rather different ways.

I've bootstrapped Rust for RISC-V on FreeBSD, it's not that bad, the issues I faced were solely porting issues, not bootstrapping issues. I've certainly not had cross-compiling issues like Drew. The awful part about porting Rust is that you need Rust bindings for every type and function in your system's libc, which is fine if your OS+libc combination is already supported (though even then 32-bit and 64-bit need some separate implementation bits), but a real pain if your OS isn't yet supported. Rust on CHERI has the additional complexity of a poor design decision in the language that defines usize as a pointer-sized integer, conflating integers and pointers, rather than providing both size_t and uintptr_t equivalents like C has; see https://internals.rust-lang.org/t/pre-rfc-usize-is-not-size-... for a discussion of this and ways to resolve it without breaking existing software on non-CHERI which, as you might expect, is to introduce a new uptr type for the rare cases when a usize holds a pointer not an offset/length/machine word-sized integer, allowing a usize to be used in its place on non-CHERI architectures (at least for existing editions) but not for CHERI architectures. Someone needs to do that engineering though and it's not a priority for us.

The only thing that gets weird is the primary thread's stack limit, since we need to construct that capability up-front with the right bounds. This means we have a somewhat arbitrary upper limit (which you could make tunable, though I don't think we do, just a #define'd constant) that should be big enough for any reasonable process and reserve all that virtual address space regardless of what the stack limit is (but it's still unmapped and not backed by anything, just a big region of "don't use this as another capability already overlaps with it"). You can then use rlimit(2) to configure your stack limit and the virtual memory subsystem will do the same as it does on non-CHERI, just you won't be able to exceed that threshold, and the capability's bounds for the stack pointer will always cover the maximum possible mapping, not the mapping you're actually using.

Everything else is the same, just "as much as your system can fit (and system policies will let you)". Which will be slightly less, because pointers are bigger, but we don't have any additional tables that impose arbitrary restrictions on what you can do.