To mitigate this kind of problem, at my company we use a library [1] that allows regularly logging which tasks are running and what file/line numbers each task is currently at. It requires manually sprinkling our code with `r.set_location(file!(), line!());` before every await point, but it has helped us so many times to explain why our systems seem to be stuck.
[1] https://github.com/antialize/tokio-tasks/blob/main/src/run_t... has set_location(), and task.rs has list_tasks()