I’m incredibly confused why profiling wasn’t even mentioned in this post.
I would highly recommend anyone troubleshooting something like this to do continuous profiling and the root cause of this type of problem becomes painfully obvious within seconds.
(Disclaimer I founded a company in this space, and we recently open sources a library for rust to do this: https://www.polarsignals.com/blog/posts/2023/12/20/rust-memo...)