The detail was right there: debugging something in security caused massive logging which caused CPU bottlenecking.

Performance is the hardest thing to integration test for. Keeping careful track of CPU/memory/network/disk load with automated alerts can help.

(Fancy systems like running a traffic replica can help, too, but at a much higher cost.)