Optimizing python with Cython | Hacker News Reader

Optimizing python with Cython

korokithakis.net

20 pointsiamelgringo7 comments

I've used Cython for some pretty dramatic results - reducing a run from 1.5 hours to about 20 minutes - but replacing the algorithm with a much more efficient one reduced the time down to about 40 seconds, at which point running it through Cython didn't make any difference anymore (and in some cases made it slower). If you are computationally bound and not I/O or data structure bound (my biggest speedup came from exchanging lists for dictionaries to eliminate linear searching - which was a trade-off in accuracy of my algorithms vs speed, but in that specific case it didn't matter) then Cython can give pretty good results. Its also useful for wrapping C libraries.

As is always the case with optimizations, a good algorithm usually goes a lot further than highly optimized, low level code.

Worth checking out pysco as well http://psyco.sourceforge.net/

Though in practice I find that if performance really matters using numpy or writing a C extension module usually works out the best.

I like psyco simply because all you have to do is import the module and do `psyco.full()` to get an idea how well it's going to work with your code. Once I feel like I've gotten the most I can possibly get out of algorithmic and data structure optimization, psyco is almost always my next stop for further speed. Often, importing and activating psyco is all the optimization I end up needing.

I would like to note though, that psyco development has been stopped for a couple years, and probably won't continue (unless a new developer thinks there's a future and takes it over). Further development is going into pypy.

Also, it can't run on 64-bit machines. If you want to run it on OS/X 10.6, you need to build your own 32 bit python.

Does this really make sense, instead of just coding your application in C/C++ or even Go? Seems like you have to rape your Python source code pretty totally.

Or how about implementing the performance-sensitive parts as native C modules for Python.

This is actually one of the ways people write C modules for python. It is a Python -> C translator tool (which the linked article doesn't make very clear).

I myself prefer sip or weave, but this is also just another way to write C that is tied into python.

I had similar speed issues in an evolutionary computing class last semester. I took it as an opportunity to learn Haskell and Scala, and noticed that in addition to getting dramatic speedups, I also had a lot more fun writing the code in the first place.

Reply on news.ycombinator.com