Change the interrupt cpu affinity to split network interrupts over multiple cores. See:
http://www.cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txtI tried enabling RPS/RFS, which to my understanding, did this; load balance the interrupt handling among multiple cores. With this enabled, I saw little to no difference in connection rate. But then again I'm guru, I might as well double check this.
Updated my little "action plan" in the original Serverfault question with this info.
This raises a few interesting cases in which the behavior of irqbalance may be non-intuitive. Most notably, cases in which a system has only one cache domain. Nominally these systems are only single cpu environments, but can also be found in multi-core environments in which the cores share an L2 cache. In these situations irqbalance will exit immediately, since there is no work that irqbalance can do which will improve interrupt handling performance.