For servers, memory is usually the long tent pole. For me it boils down to:

* Use less memory.

* Don't use a lot of threads.

* Use fewer objects.

Worst-case GC is a big issue. For just a 2 GB heap you are looking at up to 7 second GC pauses with the Java 6 collector. You can use concurrent GC but it adds significant CPU and memory overhead. Direct buffers don't live in the heap so they don't contribute to GC time.

Threads in Java are extremely heavyweight in memory, if not in CPU. A rule of thumb is about 1 MB per thread. You can make it better by decreasing the stack size and/or using 32-bit addresses.