With gdb, prints, and lots of thinking. In other words, pretty much like you'd debug one elsewhere, except instead of "run" in gdb you need "target remote <guest-gdbserver -address>".
Yes, you can add protection barriers (at least if the underlying HW supports it), but I have trouble imagining what you'd really gain. Maybe, if we assume the cloud, if you restrict the hypercalls to the "kernel", you can add some extra fences an attacker would need to go through to attack the hypervisor, but that barrier would not be at the normal user/kernel separation in a general purpose OS.