The scientific community should focus on moving to PyPy and helping to excise CPython C extensions from more scientific libraries.
That's at least a change that will benefit them.
That's at least a change that will benefit them.
PyPy is a nice way to make most Python programs 20-100% faster, often at the cost of using slightly more memory. Writing in C/C++/Cython etc, we typically see 100x run-time and memory improvements.
These extensions were written in C for a reason. If I had to write in pure Python, I'd find another language for the bulk of my work.
I don't use numpy (or really very many scientific libraries), and talking about generalities like you mentioned just isn't meaningful, but here's a really old benchmark which is (small) evidence that it's just not true that these things need to be written in C anymore: http://morepypy.blogspot.com/2012/01/numpypy-progress-report...
PyPy can address Python's slow loop performance and slow numeric calculations, in particular examples. This can help a lot. But if the problem requires you to carefully manage your data structures so that you get good cache performance, PyPy cannot help much at all.
Consider this: there are very few non-toy examples of PyPy giving order of magnitude efficiency improvements over CPython, while other languages do typically run that much faster than Python implementations.
Finally, if you don't want to talk about generalities...Consider my NLP library, spaCy: http://honnibal.github.io/spaCy/#speed-comparison . It's the fastest library with this functionality in the world, among any language.
This library is an example where you need C data structures just to make the working set manageable. You would need an enormous amount of memory just to load the models. In pure Python, the library would not be of much use to anyone --- so I would have written it in Java or C++.
You seem to be suggesting that everything is fine, and that all the C code that exists right now in the scientific Python community is justified. This is incorrect.
Needing "C data structures" (I'm guessing you mean memory management but I'd love to have a look at your library tomorrow morning) is certainly reasonable -- and there are better ways to do so than the CPython API.
My point before, which I am pushing, yes, is that the scientific Python community is using old tools, and they'd benefit from using new ones that actually would directly benefit them, which PyPy will -- CFFI is a better choice than ctypes or the CPython API for way more agreeable reasons than whether Python 3 is a better choice than Python 2, and pure-python is a better solution than both for where it's applicable -- it's unimportant where that is to me, my point is "it's more than what we currently have in pure-python".
That's a pretty tall order...
http://www.scipy.org/scipylib/faq.html#how-can-scipy-be-fast...
SciPy uses a variety of methods to generate “wrappers” around these algorithms so that they can be used in Python. Some wrappers were generated by hand coding them in C. The rest were generated using either SWIG or f2py. Some of the newer contributions to SciPy are either written entirely or wrapped with Cython.
This is a Fortran-to-Python interface, and has some support for Python 3 so far as I can tell.
In many cases, you can use something like python-bond to call PyPy from a regular CPython program: https://pypi.python.org/pypi/python-bond (or just use multiprocessing).
* Supported on more than just CPython. I.e. you can use them on PyPy as well.
* The extensions are far easier to install, since one doesn't need both a full C compiler as well as all development headers for the library installed on the server where you install the extension.
I wanted to use ctypes to wrap a C library for a recent project (for the ease of installation and development that you mention) but had to give up when it turned out to be more than 10x slower than a wrapper written in Cython, and barely faster than doing a pure Python implementation of the C library itself.