Python Debugging Techniques | Hacker News Reader

Python Debugging Techniques

aymanh.com

141 pointsedw51919 comments

The first one with 'import code; code.interact(local=locals())' is priceless.

Is there a way to attach a python shell to a running process? I saw something similar for ruby but could not find a python equivalent.

People that find code.interact useful should check out IPython* (which is an interactive Python shell with auto completion and a lot of other features).

Here is how I launch a shell for my projects (it tries to use IPython, if it isn't working it uses code.interact): http://paste.plurk.com/show/17110/

* http://ipython.scipy.org/moin/

IPython is great and I used to be a huge fan before I found Spyder (formerly Pydee.) It is somewhat a matter of taste, but when working interactively I think Spyder is one of the best shells. It is available at http://code.google.com/p/spyderlib/

Upvote for iPython, it quickened my learning of the language 10 fold. That and slime.vim is a match made in heaven!

Not that I know of. In the past I've allowed my process (which was essentially a HTTP-based app server) to evaluate commands from a secure source, then built a Python interpreter that sends what you type via to HTTP to it -- so you get a pseudo shell. At another point I've set up a signal, e.g. SIGUSR2 to essentially do the code.interact() thing like in the article -- so when you need to you can kill -USR2 the process and get into a shell. The first method lets you keep running the server while the second one freezes it while you debug it.

There are also macros for gdb floating around -- you can attach the gdb (C etc.) debugger to a python process and then examine the stack through some gdb macros, but the interactivity is limited (unless perhaps someone has built it up more). That lets you break in at any moment. See http://wiki.python.org/moin/DebuggingWithGdb

In many cases where a production process is doing mysterious things, using strace or perhaps ltrace on it can give a good hint to what it is doing, together with lsof to see what files it's reading/writing.

You might be interested in this package I wrote for one of my projects:

http://github.com/cool-RR/PythonTurtle/

Check out the directory "/src/shelltoprocess".

It's a package for connecting a GUI shell, running on wxPython, to a Python process spawned by the multiprocessing package. It's documented and MIT-licensed.

Twisted's manhole does exactly this. SSH into a process! http://www.devshed.com/c/a/Python/SSH-with-Twisted/3/

WinPdb well let you do this. It lets you start a process, and then attach a graphical debugger to it over TCP. Then you can let the process run, break and examine whenever you want, set break points, conditionals, etc. Great for debugging things like FCGIs, FUSE plugings, etc.

If you do WSGI web-applications then I can recommend following for debugging memory leaks:

* Finding and fixing memory leaks in Python's WSGI applications: http://amix.dk/blog/viewEntry/19420

This is a more general technique on debugging memory leaks using objgraph.py (that can be used in non-WSGI applications):

* Tracing Python memory leaks: http://www.lshift.net/blog/2008/11/14/tracing-python-memory-...

Talk about timely :) Thank you!

Debugging python in a server environment is a different kettle of fish though, I keep running in to situation where something goes wrong under water and there is absolutely no hint of where the problem lies. That's one of the most frustrating bits of django/python development as far as I can see.

In PHP it is a very rare occurence to get an error that does not immediately pinpoint the problem spot. In django/python if you get an error message at all chances are that it will send you off on an hour+ tour of the documentation trying to figure out what is up.

For fun have an alternate_name on a foreign key that ends on _id, you will get errors that have absolutely no bearing on the location of the problem.

oh, and another one. pycallgraph is a good way of getting your bearings, visually, when confronted with a chunk of disorganized, unfamiliar code:

http://pycallgraph.slowchop.com/pycallgraph/ http://www.ohuiginn.net/mt/2009/01/pycallgraph.html

re: logging vs. print. If you do get stuck debugging something that uses print() for all its logging, remember that in extremis you can still redirect it elsewhere:

import sys

old_stdout = sys.stdout #store the real stdout

logfile = open('/some/file', 'a')

sys.stdout = logfile #redirect output to a file

#code you're debugging here

sys.stdout = old_stdout #restore the real stdout

[Obviously using logging from the start is _far_ preferable]

Winpdb supports embedded python and remote debugging, is free and cross platform. (http://www.winpdb.org/ is down for me right now, http://en.wikipedia.org/wiki/Winpdb has some info)

Though I don't know of an effortless way to disable debug prints, you can easily redirect to a file.

import sys print >>sys.stderr, "STDERR is fun."

There's not too much that suggests using this instead of logging.

If for some reason you wanted to do this, you could just do

  import sys
  def debug(*s, sep=" ", nl="\n"):
    print >>sys.stderr, sep.join(map(str, s)), nl,
    pass

  debug("There was an error on line", 10, ":", nl="")
  debug(1,2,3,4,5, sep="-")

And then comment out the line with "print" to disable it.

Of course, by that point you may as well use the logging module.

hmm, it would be really handy if you could connect code.interact to a socket. Then when an error is raised you could telnet to the process and do a post-mortem on it...

Can you run code.interact from a django view if you've set your server to run not as a daemon?

I believe so - I use pdb.set_trace all the time in my django views (during debugging) and prefer it to code.interact

I just tried it and it works

Reply on news.ycombinator.com