Had a similar bug in a production system about 10 years ago. Indexes got corrupted after some length of time. Only in production, couldn't replicate. A distributed system as well.
We (mainly a collegue of mine) eventually traced it down to a smaller part of the code until we noticed a subtle bug.
Multi reader + multi writer and a atomic operation that miscounted the number of readers.
Took us a couple of weeks to track down if I remember right.