And another aspect we wished to demonstrate was the sheer size of permanent generation on some of the applications. I have yet to face a leak-free app where permgen > 256MB seems justified ...
Perhaps the tool you install should be showing the number of classes loaded and the size of the code in general. But, again what's in a jar file can't tell you how many classes will be loaded because many common API such as hibernate and spring generate thousands and thousands of dynamic proxies that end up being stored in memory for the lifetime of that application.
The data also doesn't show if the higher heap users tested out running a lower PermGen/Heap and ran out of memory, hence their increased size. I'd rather allocate too much memory though than run out of memory at a peak time and have one of my hospitals deal with downtime because I saved $25 worth of RAM.
Anyway, I'll read the followup but I hope it has more info. The key is you have to profile your application, tune it and your done.