They had a custom user ISA instruction to make write or read barriers cheaper.
> Azul Systems has built a custom system (CPU, chip, board, and OS) specifically to run garbage collected virtual machines. The custom CPU includes a read barrier instruction. The read barrier enables a highly concurrent (no stop-the-world phases), parallel and compacting GC algorithm.
From the paper, including some bits on the MMU you mentioned:
> Our read-barrier allows us to intercept and correct individual stale references, and avoids blocking the mutator to fix up entire pages. We also support a special GC protection mode to allow fast, non-kernel-mode trap handlers that can access protected pages.
> Having the read-barrier implemented in hardware greatly reduces costs. In our case the typical cost is roughly that of a single cycle ALU instruction.
> The hardware TLB supports an additional privilege level, the GC-mode, between the usual user- and kernel-modes.... Several of the fast user-mode traps start the trap handler in GC-mode instead of user-mode.
> The TLB is managed by the OS in the usual ways, with normal kernel-level TLB trap handlers being invoked when normal loads and stores fail an address translation. Setting the GC privilege mode bit is done by the JVM via calls into the OS. TLB violations on GC-protected pages generate fast user-level traps instead of OS level exceptions.
> The hardware supports a fast cooperative preemption mechanism via interrupts that are taken only on user-selected instructions, allowing us to rapidly stop individual threads only at safepoints. Variants of some common instructions (e.g., backwards branches, function entries) are flagged as safepoints and will check for a pending per-CPU safepoint interrupt. If a safepoint interrupt is pending the CPU will take an exception and the OS will call into a user-mode safepoint-trap handler.
Then read starting with section 3.3 Hardware Read Barrier for the details.
See also C4: The Continuously Concurrent Compacting Collector by Gil Tene, Balaji Iyengar, and Michael Wolf for how they moved this to x86 hardware: https://www.azul.com/files/c4_paper_acm1.pdf