A guide to analyzing Python performance | Hacker News Reader

A guide to analyzing Python performance

huyng.com

79 pointsrbanffy7 comments

This guide is a nice start, but it misses the easiest way to analyze Python performance -- using the %time, %timeit and %prun magic commands in IPython, the better python command line interface (and which also has a killer web notebook interface).

The line and memory profiler modules he mentions can be installed as IPython extensions, too, and you don't even have to modify your code with decorators to make them work.

Good. Having spent months (a while back) analyzing where memory leaks were occurring in a fairly big code base I can say that it is no fun without good tools. In that case the issue was that the underlying homegrown library was based heavily on SQLAlchemy. Our SQLA objects referenced each other and when we freed the session, they were not decoupled automatically.

Also beware crappy C extensions. After much debugging I found a quad tree implementation that leaked like a sieve. From what I understand these tools will not help with that, though it is rare.

>Also beware crappy C extensions. After much debugging I found a quad tree implementation that leaked like a sieve. From what I understand these tools will not help with that, though it is rare.

As someone who has had to find and resolve memory issues with C extensions, I would suggest using Valgrind [0]. It is not a python tool, but (after compiling Python with to play nice with memory debuggers), it works well for debugging memory in C-extension. It basicly runs your program in a VM, where it monitors memory access and reports leaks and other potential errors (such using unitialized, unallocated or deallocated memory) along with a stack trace.

[0] http://valgrind.org/

Yup. Valgrind is the bees knees when it comes to debugging any C program. I do not remember if I used it or not for this particular purpose, but it definitely would have helped.

memory_profiler [0] mentioned in the post actual can detect memory leaks in compiled C extensions as it uses the OS through the psutil module to measure the aggregate memory usage of the Python process rather than asking the Python runtime itself. It can therefore take into account memory that was not allocated by the Python malloc.

[0] https://github.com/fabianp/memory_profiler

There is also the nice CProfile [0] that comes with the Python Standard Library. It does more or less what the mentioned line profile does, with the bonus that you don't have to install another package.

http://docs.python.org/3.3/library/profile.html

There is also Guppy/Heapy, which I like in particular because you can spawn a monitor that you can connect to externally. Sadly the docs aren't very complete.

Reply on news.ycombinator.com