The problem was subtle -- some part of a program was writing to a memory location it shouldn't be, making random memory locations 0.

We had a custom moving GC. Fixing the bug involved changing the GC so, we used mmap to map an anonymous buffer storing the GCed objects into many memory locations, and each time a memory block was allocated:

1) Choose a random copy of the memory.

2) Mark all the other blocks as neither readable or writable.

3) Make all references to GC memory we were currently tracking point to that block.

This meant anyone who was keeping a pointer to GCed memory we had lost track of would point to a now protected block of memory, and as soon as they tried to read or write through it would cause a segfault.

This made the program run slower.. so much slower the startup, which used to be about 3 seconds was now about 6 hours. However, it found the bug (and a whole bunch of other GC bugs as well).