All of the operations in the plots are being calculated using tiny arrays. Even 128 elements is pretty small. Additionally, I'd be interested to know how they compiled NumPy, and which BLAS implementation they linked against.
So what the graphs end up highlighting is that the CPython (NumPy?) has decent overhead on tiny calls that don't really matter for gross performance. As you say, 128 elements is tiny: that's where I'd consider starting a graph, not ending it. At 4 elements, you're looking at about 40 clock cycles to do a simple vector-add in the scalar case (I think x86 L1 cache hits are ~3 clock cycles and there are two load/store units; if these numbers are wrong, my estimate is off). This also means that there's absolutely no measurement of the overhead of the JIT autovectorizing operation (!), which is quite significant because the usual resistance to adding autovectorizers in JITs [1] is that they're way too slow for the speedup they give.
[1] As far as I'm aware, the only other JIT of note that includes an autovectorizer is the Hotspot JVM. The approach taken by other people (e.g., .NET CLR) is to effectively expose a SIMD primitive in the bytecode and get compilers to target those via static autovectorization instead of autovectorizing at runtime.
128 elements is still useful for when you have an inner loop that has a small vector operation. Most of the time this is not the case though.
I definitely agree that overhead should be included in the article, you wouldn't expect to see it in those graphs, because it's a once off, right?
Vectors are only fixed-width, so even though it is an "embarrassingly parallel" problem, you still only expect it to asymptote towards a fixed speedup. Moreover, the bigger the matrix, the more likely you are to see cache size and memory bandwidth effects in your performance numbers, meaning the SIMD could be less of a win in the limit.
There are a lot more details sprinkled across several pages: http://morepypy.blogspot.mx/2013/11/numpy-status-update.html http://pypy.org/numpydonate.html