I use gdbserver to attach gdb to a process on the target machine, running gdb on my dev box where the sources are. In a previous life, I would use a similar capability to debug code in an embedded target through a serial port, with a tiny stub in the target program to manipulate target memory and execution. Nowadays you do that through a USB port.

When I set a breakpoint, I can make it conditional on an expression run against program data, and have it run a series of debugger commands at each stop, and then maybe continue.

Scrolling back through the transcript to see how values have changed at each stop, and running expressions on e.g. differences between such values.

All simple things.