In theory, you can make conditional breakpoints very fast using an in-process agent. For GDB (for instance) this gives the ability to evaluate conditional breakpoints within the process itself, rather than switching back to GDB: https://sourceware.org/gdb/current/onlinedocs/gdb.html/In_00...
I've always found the GDB documentation to be a bit vague about how you set up the in-process agent but I found this: (see 20.3.4 "Tracepoints support in gdbserver") https://sourceware.org/gdb/current/onlinedocs/gdb.html/Serve...
When we implemented in-process evaluation of conditional breakpoints in UDB (https://undo.io/products/udb/) it made the software run about 3000x faster than bare GDB with a frequently-hit conditional breakpoint in place. In principle, with a setup that just uses GDB and in-process evaluation you should do even better.