Thanks for the suggestions, here's some clarifications:
* I clarified what hardware I've tested in a comment (see below).
* I've run tests with my server ranging from 10 sec up to 10 minutes. They're consistently bad unfortunately.
* Interesting what you say about dom0/domU. I'm no Xen guru, but the culprit is probably something like that. I've been using a backlog of 1024 for the server tests (set both in Java land and sysctl.conf). The netperf are all defaults, both in terms of run time and backlog. Was actually trying to monitor the backlog somehow, but I'm not sure that's even possible in Linux?
* The server isn't doing anything disk IO-bound so this shouldn't be the case.
* syncookies seems like a good idea, I will definitely try that along with a lot bigger backlog and see if it makes any difference. I'll also try and see if netperf can be tweaked as well to provide a better, isolated test case.
Writing this off as Xen overhead would be such a shame, virtualization should not cause this much overhead. I'll continue investigating!