Just drain the traffic (make the load balancer temporarily not send traffic to it) and attach a debugger. I hope your load balancer has that feature because health checking also needs it.
ah, that's probably a missed wakeup problem! Much nastier. Still seeing where your thread is blocked might hint you to where the missing signal should have been.
I didn't mean literally a single thread, just that not the whole application is hang.
So I actually had this exact issue where we suspected a deadlock, but the application was limping along and we couldn't justify attaching a debugger. I managed to confirm my suspicions with judicious use of perf and /proc/<pid>/task/<tid>/wchan .