> Is unoptimized code really that bad?

Most application code is far worse and stalls (sadly - almost literally) every few instructions for L2 / memory.

> (Code is small relative to data, most of it should be in cache most of the time)

Not if your application is dragging two and a half invasive runtimes around.