Was it the boolean check that degraded performance or was branch misprediction inside a nested loop the root cause?

The solution you came to seems quite elegant but I'd be curious to see metrics with the debug_on check moved outside both loops.