They're optimized for single-precision linear algebra. If you need double-precision all those optimizations go out the window and you're left on your own.
Not true at all on modern X86. You just get half the FLOPS by using double precision instead of single precision -- 16 double precision FLOPS per core per cycle. Which is very good considering there's also twice as much data to process.
You could rather say X86 CPUs are highly optimized for double precision performance.
I still wouldn't say that they're highly optimized, though.
"In most cases, double precision calculations take no more time than single precision."