> I don't think the garbage collector is to blame. During the worst times, when the JVM almost maxed out 90 CPUs, the GC was only using 3s of CPU time per minute.
The graph linked only shows 3s in young gen GC - you should check the time spent in the old gen GC too.
You can get loads of time spent in GC even without running out of memory - running out of other resources like file handles or network connections will also trigger a full GC in the hopes of freeing some up.
If you've got 1000 file handles available, one process that uses 100 per second and doesn't leak, and another that uses 1 per hour and leaks it, after 900 hours everything will look fine, then after 1000 hours you'll run out - and the symptoms will manifest in the first process, not the second.
Admittedly, there's a text file of jstats output linked which doesn't show any full GC happening, so maybe this is nothing...