You should check if you have a listen queue overflow and syncache buckets overflow in netstat -s.

For FreeBSD you should also check if there are packet drops in sysctl net.inet.ip.intr_queue_drops