I was going to say for people who are going to lengths like kernel bypass networking to get these trades out faster I wouldn't be surprised if the key stuff is in assembly.
I would be. Manually written assembly is not necessary better than what a compiler would generate and with super scalar pipelined architectures, seemingly better assembly can turn out to execute in the same amount of time. The things that matter are algorithmic.
I think you assume compilers are doing more work than they actually are. Processor models in the compiler are generally fairly simplistic and even ICC can easily be beat by hand tuned assembly.
Where compilers shine is allowing broad process-wide optimization.