I occasionally contemplate that, if I were designing an OS meant to be sort-of-immutable (like Incus OS or Fedora Silverblue etc or MacOS), I would probably build it like this:
The main filesystem is verified and immutable. Everything that isn't configuration or the user-controlled payload is genuinely read-only, and the system will even cryptographically verify it on boot or first use. You cannot modify /bin/bash, etc.
If you want to test a modification, you can configure an overlay, and you can boot with that overlay live. You can configure the overlay to also be immutable or you can make the overlay mutable. But the choice of booting into the overlay is controlled by code that cannot by overlaid, so you can always turn the overlay off no matter how much you screw it up.
The user may get root access, but if your system is remotely attested or uses a TPM or such for security, then that policy will find out if you do so before you can do anything as root. So you can shell in and attach a debugger to a system service, but you cannot do that and also pretend to your orchestration tools that you have not done so.
The default configuration is mostly empty. When you change a default, you are not modifying the middle of a giant plist where no one will ever understand what happened. You only create new configuration, and deleting it is just fine.
The result would, I think, give system owners plenty of ability to hack on their own systems, but they could also unhack their systems easily. There are very few systems out there with both of these properties right now...