We're thinking of giving up on scaling above 16gb/jvm and going to something crazy like 16x2gb jvms. Treat them like the worlds biggest mongrel and let the loadbalancer sort it out.
Mind emailing me your config? You should be using a CMS collector. -XX:+UseConcMarkSweepGC. Also try reducing CMSInitiatingOccupancyFraction until things stabilize. I've never had to drop it below 70%. Finally, ALWAYS make sure Xmx and Xms have the same value so the JVM doesn't try to resize itself.
That (and a truckload of trial & error) led me to this config: http://pastie.org/pastes/7130367/text?key=s8jajyejyhgz7x303s...
You're probably right, I probably could just keep dialing up xms/xmx/xmn, but even then what exactly to watch for in the gc log and how to do it tools-wise is a dark art. Exactly what field in what line to grep for and be mindful of when is a constant source of debate/confusion, and once you get up to a few hundred req/s visualgc just can't keep up so the graphs stop being useful.
As you can see from my config link there I've tried throwing money at the problem but that didn't do it either. I'm at the point where I can hire an engineer to truly master this, or go bootleg parallel.
Understanding the gc logs can be difficult. gcviewer is pretty good though at figuring it out. Always remember that garbage collection is slowest whenever the JVM has to copy objects from one generation to another or between survivor spaces. The logs print the sizes of the young gen, old gen, and survivor spaces. Sometimes after a collection you can see the sizes of these change dramatically - that means a big object has moved around. You can either change your code to use big objects less often or hold onto them for a shorter time or try to increase your survivor space size or new gen size.
A major, major, major thing you are missing is -XX:+UseParNewGC. With the CMS collector, your new gen collection can be done in parallel, and the parallelization scales very well with the number of cores. Without ParNewGC, the service I work on would take over 1 second to garbage collect. With ParNewGC, it only takes 50ms.
Also, more concretely, this is the most frequent type of entry in my gc log: http://pastie.org/pastes/7142036/text?key=dkwvlvaaysk63rcajb...
I (perhaps wrongly?) interpret the ParNew there to imply that's the young gen collector in use.
[1]: http://www.azulsystems.com/products/zing/virtual-machine