Thoughts on porting NumPy to PyPy
technicaldiscovery.blogspot.com
technicaldiscovery.blogspot.com
Also, his point that Cython (http://cython.org) tends to be ignored in the broader discussion about performance computing in Python is especially flagrant when you consider how it's revolutionized the way that scipythonistas (myself included) speed up their code over the last 2-3 years.
An initial port seems like the only way forward from the point of view of the pypy team. I think it is unrealistic to expect the pypy team to take on the work of changing numpy so that it is more friendly to alternative implementations. I would certainly expect them to be involved in the discussion though.
The article also seems to miss that there is work ongoing to bring pypy support to Cython.
I think this is a promising idea for the future of array-oriented computing, as it can make use of one more level of parallelism / scaling than the current Numpy paradigm, which is limited to one operation at a time and the user providing the ordering of operations.
I can especially relate to this, even though I'm 26. Admittedly, I work in a microcosm though, where I'm the youngest, least educated and least experienced in my group, despite having a BS in Physics and 4 years research in particle astrophysics.
Luckily we do a little less with matrix operations, but I agree that Python, specifically PyPy, has so much potential to be the scientific computing standard and that the python community should really push for it. In a world with less python, there's so much pain involved on a regular basis setting up software like ROOT, switching between libraries of FFTs or plot libraries, installing Octave or Matlab to work nice with some bash script and dealing with OS discrepancies in getting someone else's code to run. It sucks.
If PyPy can displace that with near-C performance, the world would be a much better place.
I suspect the real problem is NumPy's type system implementation, since data-types are not visible as different Python types.
a * b
is elementwise but
a %* % b
is matrix multiplication. If you write down a complicated linear algebra expression, something like
A.T ! (B.T ! C ! B).T ! A
would be a lot friendlier to scientists than the current:
dot(A.T, dot(dot(B.T, dot(C, B)).T, A))
You might just say "well suck it up" but I've got to say that doing linear algebra in Matlab is a lot easier because the linear algebra that I do with pen and paper looks pretty much exactly the same as the corresponding code. On the other hand, Matlab is super clunky compared with NumPy at doing APL-style array processing with broadcasting operations, etc. In my work I tend to do more of the latter and less of the former but whenever I implement something with a lot of matrix multiplications it takes me a lot longer in Python to get things right.
Anyway, the point is: non-scientific Python folks need to take a walk in our shoes to gain an understanding of the challenges we face on a ongoing basis.
I'm having a hard time understanding your last statement. NumPy data types (dtypes) simply tell the ndarray how to interpret the block of data associated with it (the # of bytes per item, shape, and strides).
with numpy.doing_matrix:
matrixy_multiply_goodness = m1 * m2
However, Python is a general purpose language, and scientific computing is a single domain. It is used in many different domains. Consider that there are many changes that individual communities would like, and if Python granted all of those requests, the language would be a mess. That's the challenge in designing a general purpose programming language.
If we define "general purpose" as "works well in as many domains as possible", then I'd argue that the correct response to an easily-remedied weakness in an important domain should be to address that weakness.
As you imply, the challenge here is to decide which requests to grant and which to deny, but the mere fact that a request is "domain-specific" (ignoring, for the moment, the questionable idea that _linear algebra_ is domain-specific) should not be enough to rule it out.
(As an aside, scientific / numeric Python is, I believe, one of its two or three most important application areas, and perhaps the most important historical factor in its success: that community has been championing Python since the days when people were using Perl for the web, or for anything else. Notice, for example, that Travis mentions working on SciPy in 1999---and the original "Numeric" package was written in 1995.)
I think this is shortsighted to the point of ignorance. "Scientific computing" here really means "performance-critical numerical computation on regular arrays". Basically all of the new things that people are doing with computers in the last five years and the next five years — machine learning and other statistics, software-defined radio, audio synthesis, real-time video processing, cool visual effects, speech recognition, machine vision, and 3-D rendering, and arguably Bitcoin — consist largely of performance-critical numerical computation on regular arrays. It's what GPUs are for. Five years ago, Numeric or NumPy was probably the best way to do that for a wide range of things, although a lot of people still use Matlab instead, and R deserves at least a mention. Today it's not clear. Five years from now there will be something much better than current NumPy, and it could be a better version of NumPy or it could be R or Matlab or Octave or something.
In short, "scientific computing" is not a single domain, but a set of capabilities increasingly important in many different domains.
I am, however, self-aware enough to recognize that what I care about is a subset of what everyone in computing cares about. Scientific computing may be important in multiple places, but it is still a single domain. An important domain, sure. But Python is still used in many places were such concerns are not important. Arguments about the utility of number crunching on dense vectors and matrices are great. But the attitude I've seen in this thread is "the domain I care about is so important that my concerns should be elevated above the concerns of other domains." That is not going to fly when it comes to changing a general purpose language used in many domains.
In the 1960s, there were those who claimed that the benefit of recursion was not worth the complexity it added to programming languages, except in certain special domains. In the 1970s and 1980s, we had the same argument about depth-first search — SNOBOL's and Prolog's backtracking feature. It turned out that the proponents of recursion were right, and the proponents of backtracking were probably wrong. (Although the backtracking feature of SNOBOL's child Icon directly inspired Python's generators, although implementationally they're maybe more similar to CLU iterators.)
Now, when it comes to matrix and vector manipulation, should we imagine it as a single domain, or as a feature that's useful across a wide range of domains?
I'm arguing for the latter.
The last comment about dtypes is that the size (shape in NumPy) should be part of the type; certainly element-type should be. A 1-D vector of double should not be the same Python type as 3-D array of characters or a 500x600 matrix. This creates havoc in a dynamic language. I once spent two days tracking down a bug caused by "*" multiplying two arrays instead of two matrices when I started using NumPy. Perhaps size is too much for a dynamic language to be part of type, but surely dimension and element-type should be reflected in Python type. It absolutely does not help that the documentation is littered with type-objects and object-objects; I appreciate this is how C implementations are, but for a beginner NumPy users, it is more than a little confusing.
I have a lot of respect for people who designed and built NumPy; for multi-dimensional arrays I don't see a better approach; but too much of influence of C implementation details seep into Python interface, types and operator overloading are only the beginning of the problems. I am having second thought about the suitability of dynamic languages for large scale, high-performance computing. Type annotation could help a lot but I see there is exactly zero interest in that for NumPy.
As a general rule, "Special cases aren't special enough to break the rules", but like everything in Zen it is a balance. If a very large community of Python users (and the scientific users ARE a large and important subset of Python users) say they would benefit significantly from this change, then perhaps this is the exception -- especially since the "cost" (in additional complexity) is fairly small.
A more plausible scenario would be both CPython and PyPy using numpy-py to call into numpy-c and numpy-pypy respectively.
I'm on Ubuntu Ocelot & Firefox 7.
nevermind: I'm seeing it on various websites that use ligatures. Possibly because I'm half-way through an Ubuntu upgrade.