I suspect the assembly generated by the compiler is pretty fancy.
Edit: Just briefly looking at the objdump again, gcc is using a lot of scalar variants of instructions, instead of vector. Maybe this is giving the CPU better hints?
Maybe my code would run better on a more modern CPU (while still constraining the original C program to SSE3 only) ?