If I remember correctly, the IEEE 802.3an line coding overhead is around 2.6ms for each time you switch between SPF+ and RJ-45. That's in line with my measurements.
SPF+ switch to OM3 fiber to SPF+ PCIe card => 0.2ms
SPF+ switch to RJ-45 cable to RJ-45 PCIe card => around 3ms