Debugging a Mixed Python and C Language Stack | Hacker News Reader

Debugging a Mixed Python and C Language Stack

developer.nvidia.com

152 pointsquasiben23 comments

When developing PyTorch, we also run into a lot of mixed Python/C++ language situations. We've recently been experimenting with in-process 'combined' Python/C++/PyTorch 2.0 stack traces to make it easier to understand where code is executing (https://dev-discuss.pytorch.org/t/fast-combined-c-python-tor...).

Funny, I was going to post this, but the horse's mouth is even better ;)

GDB can decode Python stack traces as Python code, as long as you have have python-gdb.py from the Python source distribution in the same place as your Python executable.

Wow, that's pretty amazing. I wonder how it's implemented, and if there are any tutorials on implementing something similar, for programming language designers/creators.

Theres also Py Spy, a profiling tool that can generate flame charts containing a mix of python and C (or C++) calls.

https://github.com/benfred/py-spy

It's worked really well for my needs

py-spy also makes it very easy to scrape from existing processes, which is invaluable if you are e.g. ever trying to diagnose a deadlock

I think it uses gdb's frame filter API: https://sourceware.org/gdb/onlinedocs/gdb/Writing-a-Frame-Fi...

GDB itself is fully scriptable in Python:

https://sourceware.org/gdb/current/onlinedocs/gdb.html/Pytho...

From https://github.com/jupyterlab/debugger/issues/284 en ENH: Mixed Python/C debugging (GDB,) #284

> "Users can do [mixed mode] debugging with GDB (and/or other debuggers) and log the session in a notebook; in particular in order to teach.

> Your question is specifically about IDEs with support for mixed-mode debugging (with gdb), so I went looking for an answer:

> https://wiki.python.org/moin/DebuggingWithGdb (which is not responsive and almost unreadable on a mobile device) links to https://fedoraproject.org/wiki/Features/EasierPythonDebuggin... , which mentions the py-list, py-up and py-down, py-bt, py-print, and py-locals GDB commands that are also described in * https://devguide.python.org/gdb/ *

> https://wiki.python.org/moin/PythonDebuggingTools Ctrl-F "gdb" mentions: DDD, pyclewn (vim), trepan3k (which is gdb-like and supports breaking at c-line and also handles bytecode disassembly)

> Apparently, GHIDRA does not have a debugger but there is a plugin for following along with gdb in ghidra called https://github.com/Comsecuris/gdbghidra [...] https://github.com/Comsecuris/gdbghidra/blob/master/data/gdb... (zero dependencies)

> https://reverseengineering.stackexchange.com/questions/1392/... lists a number of GUIs for GDB; including voltronnn:

>> There's Voltron, which is an extensible Python debugger UI that supports LLDB, GDB, VDB, and WinDbg/CDB (via PyKD) and runs on macOS, Linux and Windows. For the first three it supports x86, x86_64, and arm with even arm64 support for lldb while adding even powerpc support for gdb. https://github.com/snare/voltron

> "The GDB Python API" https://developers.redhat.com/blog/2017/11/10/gdb-python-api... describes the GDB Python API.

> https://pythonextensionpatterns.readthedocs.io/en/latest/deb... may be helpful [for writing-a-c-function-to-call-any-python-unit-test]

> The GDB Python API docs: https://sourceware.org/gdb/onlinedocs/gdb/Python-API.html

> The devguide gdb page may be the place to list IDEs with support for mixed-mode debugging of Python and C/C++/Cython specifically with gdb?

These days, we have CFFI and Apache Arrow for C+Python etc.

https://wiki.python.org/moin/DebuggingWithGdb lists the GDB debugging symbol packages for various distros

FWIU Fedora GDB now optionally automatically installs debug syms; from attempting to debug TuxMath SDL with GDB before just installing the Flatpak.

https://developers.redhat.com/articles/2021/09/08/debugging-... (2021) describes more recent efforts to improve Python debugging with c extensions

What I would love is also to be able to set python breakpoints from gdb. And integration with rr, so I could reverse-continue to said breakpoints. And ponies.

GDB does so much cool. It just sucks that it's missing a high quality front-end. I know there are a bunch out there, but they all seem pretty janky and get confused a lot; I just want a really serious group of people (some Red Hat people for example) to make a high quality native debugging app which properly speaks the GDB protocol and handles all the edge cases.

(And no, I don't want it integrated into an IDE.)

This has been packaged as Tools/gdb/libpython.py lately.

That reminds me of the time I debugged some code (also neural network) that was in both Java and C++ and I was able to attach both gdb and jdb to the same process but had to disable the segfault trap on gdb because the jvm segfaults all the time in normal operation.

Which reminds me that with IntelliJ you can get a really nice combo for debugging Java/C/C++ code: https://youtu.be/8Cjeq4l5COU

As a C++ developer, I'm struggling to understand how and why segfaults would ever be part of normal operation. In my mental model, the presence of a segfault means that a program has gone so far off the expected path that no guarantees can be made about its state whatsoever, so the only safe thing to do is to let the program crash. Is there a reason why the JVM regularly segfaults?

A pretty good investigation into this matter - "segfaults in Java (redux)":

https://www.lukeshu.com/blog/java-segfault-redux.html

Read the prior article for some context, too ..

Segfaults are unexpected and shouldn't happen in a conforming C++ program that doesn't evaluate operations with undefined behavior.

But in non-portable C or C++ program doing manual memory management with OS primitives targeting a specific OS the conditions for segfault are well documented, and you can also rely on the programs behavior when that happens (the OS raises a signal for you, that you can handle).

There are not many programs that should be written this way, but I assume the JVM might fall into this category. I'm still not sure if handling page faults this way in regular operation is the best strategy, but I would worry about performance more than correctness.

I think Java catches the segfault and throws a NullPointerException or something like that.

An inspiring, albeit daunting, write-up for someone like me, who has been a python coder for over 10 years, but who has never professionally coded in C or C++. I'd be pleasantly surprised with myself if I could one day debug like that.

Also somewhat depressing that, yet again, the GIL was to blame, and that after all that impressive investigatory work, the fix was (spoiler alert!) "rewrite the offending function in C".

I predominantly develop in Python but somewhat frequently peer into the C++ and CUDA underlying Pytorch. I am not a competent C or C++ programmer but often am able to debug and fix bugs that originate in the lower levels of the stack. That ability developed just through exposure and focused prodding when I found issues, but I don't think it has much relation or translates to being a professional coder in C/C++/CUDA. Coding and debugging are different skill sets IMO.

It's unfortunate that no one thinks about the debugging experience with anything they build today. It's always a "strap yourself in" experience like in the post.

I dream of an IDE that is one-click, full-stack, local debug.

I dropped some ideas here: https://news.ycombinator.com/item?id=35650253

Awesome to see some more Excalidraw diagrams on the front page, feels like it's really taking off.

Reply on news.ycombinator.com