Debugging with GDB | Hacker News Reader

Debugging with GDB

sourceware.org

195 pointsingve75 comments

One absolutely essential companion to gdb is 'rr' [0], it's a 'post mortem' debugger, but unlike a core file, you can replay the action, in reverse if necessary. You can reverse-cont, reverse-step create watchpoints, run forward again etc etc. It's absolutely amazing!

[0]: https://rr-project.org/

The best thing about time travel debugging is that you effectively get an instruction-perfect reproducer for the problem you're looking at.

Even if the problem only occurs one time in a thousand, if you have recorded it then you can review it in arbitrary detail as many times as you need. You can even send the recording to another developer for their feedback.

Probably the second best thing about time travel is watchpoints[0] + reverse-continue[1] - combining these two features, you can answer "why is that value there?" incredibly quickly.

Reverse execution + watchpoints is so powerful it can speed up the diagnosis of memory corruptions, race conditions, algorithmic bugs, etc from days/weeks to hours.

[0] https://sourceware.org/gdb/onlinedocs/gdb/Set-Watchpoints.ht...

[1] https://sourceware.org/gdb/onlinedocs/gdb/Reverse-Execution.... - GDB has an implementation of reverse debugging but larger projects like `rr` (https://rr-project.org/) or `udb` (https://undo.io/solutions/products/udb/ - which, disclaimer, I work on) exist to do this with higher performance on complex, real-world programs.

(edit: format links better)

Time travel debugging https://en.wikipedia.org/wiki/Time_travel_debugging :

> Interactive debuggers include the ability to modify code and step forward based on updated information.[4] Reverse debugging tools allow users to step backwards in time through the steps that resulted in reaching a particular point in the program. Time traveling debuggers provide these features and also allow users to interact with the program, changing the history if desired, and watch how the program responds.[5]

https://github.com/rr-debugger/rr :

> System requirements: Linux kernel ≥ 3.11 is required (for PTRACE_SETSIGMASK).

> rr currently requires either:

> An Intel CPU with Nehalem (2010) or later microarchitecture. [OR] Certain AMD Zen or later processors

Is there an rr-like reverse time-travel debugging tool for ARM64/aarch64?

Are GUIs like Voltron and Ghidra helpful for gdb and/or rr-like traces?

rr works in principle on AArch64 though it requires an extremely recent CPU and the rest of system userspace to be compiled correctly.

Any gdb front end can probably be made to work with rr. Our current project is (shameless plug) https://pernos.co/ which provides a super-advanced GUI for working with (x86) rr traces.

"Support ARM" #1373 https://github.com/rr-debugger/rr/issues/1373

rr (software) https://en.wikipedia.org/wiki/Rr_(debugging)

Record and replay debugging https://en.wikipedia.org/wiki/Record_and_replay_debugging

/? site:github.com inurl:awesome https://www.google.com/search?q=site%3Agithub.com+inurl%3Aaw...

> Is there an rr-like reverse time-travel debugging tool for ARM64/aarch64?

Aside from `khuey`'s comment (https://news.ycombinator.com/item?id=30779913) that `rr` can now work on ARM64, we at Undo (https://undo.io) have an ARM64 port under routine internal testing as we expect it to be important in the future.

GDB's built-in record/replay is also supposed to work on ARM64 - https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and...

Last I heard, on x86, the performance and memory overheads of that implementation were quite high but if it solves your problem maybe that's fine!

For Windows folks, WinDbg also supports Time-Travel Debugging:

https://docs.microsoft.com/en-us/windows-hardware/drivers/de...

Took me some time to get familiar with it but GDB is well documented and fulfills its duty. It looks ancient but it is powerful.

Do not underestimate the power of the print command! And the advanced stuff like revers-debugging, multi-threading (all-stop mode - which is the default - and the non-stop mode), binary manipulation (yep, it is possible), and remote debugging. Nowadays accompanied by stuff like debuginfod. What still need to try is the TUI modes because I assume it will improve my personal experience once I find a way of using it.

I've ended up writing all stuff useful for me onto paper, later a markdown and I can recommend everyone doing similar. GDB's learning curve is not like VIM (which is either a wall or a rocket - both looking similar on paper) but it is going upwards.

Today's highlight is "Why do I've to set up all my breakpoints again? You need to save them like an IDE":

https://sourceware.org/gdb/onlinedocs/gdb/Save-Breakpoints.h...

  > I've ended up writing all stuff useful for me onto paper, later a markdown

Would you like to share? :)

> It looks ancient but it is powerful

It is powerful because it is ancient.

> Do not underestimate the power of the print command!

Still waiting for it to print the contents of a QList or QString, or pull actual data in cases where I see <optimized out>.

As I understand it, you can plug in the qt pretty printers from https://invent.kde.org/kdevelop/kdevelop/-/tree/master/plugi...

How do I use these pretty printers outside of KDevelop? When I source that file from my global .gdbinit, I get the classic Python relative import error:

  ModuleNotFoundError: No module named 'qt'

Changing KDevelop's gdbinit to `from .qt import register_qt_printers` results in:

  ImportError: attempted relative import with no known parent package

The problem is that sourcing KDevelop's gdbinit doesn't let Python import packages relative to that file. Is there an alternative way to do this?

I've never actually used these, I just knew they existed. I found them via DuckDuckGo, in the github repository of qBittorrent. They have a step-by-step guide for using them, without mentioning kdevelop:

https://github.com/qbittorrent/qBittorrent/wiki/Setup-GDB-wi...

The best solution I'm aware of for <optimized out> is to choose your preferred flavour of time travel debug and then step backwards until the variable comes back into scope.

This will do the trick, so long as the compiler hasn't completely elided a value.

It feels like there's scope for the DWARF debug info to make optimised variables debuggable normally but that's probably a pretty hairy problem.

On a similar note - add `set history save on` to your `~/.gdbinit` and retain command history between runs of GDB (just like with your shell).

https://sourceware.org/gdb/onlinedocs/gdb/Command-History.ht...

to my surprise, vim has built-in support (:help termdebug) for gdb. It's not as chiselled as some GUI debuggers but still usable and makes it (visually) easier to debug in terminal.

Accidently I also moved my mouse cursor over a variable while stopped at a breakpoint, in a terminal, over ssh, and there is a hovering balloon popping up with the current state of that variable. Mind blown.

I still struggle with GDB but my excuse is that I seldom use it.

When I was studying reverse engineering though, I came across a really cool kit (which I've yet to find an alternative for lldb, which would be nice given: rust)

I'd recommend checking it out, if for no other reason than it makes a lot of things really obvious (like watching what value lives in which register).

https://github.com/hugsy/gef

LLDB's closest alternative to this is called Venom, but it's not the same at all. https://github.com/ovh/venom

I was using gdb-dashboard: https://github.com/cyrus-and/gdb-dashboard seems kind of like gef.

In a past life I used Voltron[0] with lldb. Depending on your use case, it might be enough? Watching what value lives in what register works at least (or at least worked last time I used it). It's designed around having things in separate terminals, so you'll need tmux/screen/tiling window manager to get a similar view to gef.

[0] https://github.com/snare/voltron

Disclaimer: I work for undo.io (see my username) on time travel debugging. We use GDB as the front end but the engine underneath is proprietary.

The GDB Documentation is really nice (as GNU project docs tend to be) and I've made lots of use of them. The fact that they've also documented things like the serial comms protocol is a level further still.

Some related stuff we (Undo) have produced about GDB and might be interesting to the audience here...

Articles about less-known parts of GDB functionality: https://undo.io/resources/gdb-watchpoint/

Greg Law's quick talk on little-known features of GDB: https://undo.io/resources/cppcon-2015-greg-law-give-me-15-mi...

Greg's bigger talk on GDB: https://www.youtube.com/watch?v=-n9Fkq1e6sg

I learned a lot of debugging using GDB from Greg Laws CPPCon talk https://www.youtube.com/watch?v=-n9Fkq1e6sg. With that being said, one of the things I've heard is that debugging on Linux is a poorer experience than it is on Windows. I believe that RAD Game Tools tried to create a debugger for Linux that was better, but the project failed for some reason. I hope with the increased interest in Linux, GDB or another debugger can be improved enough to address the issues people have historically had with it

Interesting, the last time GDB hit the front page the consensus seemed to be it was far ahead of all the competition. Having never seriously used a debugger it seems like it’s an opinion to me.

I found the link http://www.radgametools.com/debug.htm. Though it looks like the project has been abandoned https://github.com/ValveSoftware/vogl Taken from their FAQ on that page about "Why make a new source-level debugger?"

> To put it bluntly, debugging on Linux is just really bad. We believe it is the biggest roadblock to great software for that platform. Don't get us wrong - debugging on Windows isn't great either. But the standard Visual Studio debugger is still much better than anything on Linux.

> So we want to make Linux debugging as good as Windows debugging. And then we want to make both better - much better. There are so many better debugging tools that we can imagine having, and yet even today's best debuggers provide barely any functionality that wasn't in debuggers twenty years ago.

> Also, we think there's a lot of value in having one great debugger that you can use on every platform. RAD develops software for roughly 18 platforms now. A big part of the friction of a new platform is dealing with different toolsets.

> Compilers, linkers and runtimes are all hard to deal with, certainly, but you typically finish setting those up in a few days or weeks. The debuggers, on the other hand, are tools you have to use every day for as long as you develop on that platform!

I guess it's just really hard to do this stuff

Thanks for the link! I’ve been diving into C++ for my personal project so I’m getting interested in debugging (my old C days were low level programming and mostly print statements.)

I've used gdb exclusively for over 10 years now (been working on Linux, worked on Windows prior to that). gdb may well be more powerful overall, but for common use cases (setting breakpoints, stepping in/out, examining values) I still miss the Visual Studio debugger. It's the only thing from Windows I miss.

Interesting. Got any tips (or articles) for someone who is beginner level with debuggers (I wrote some C almost a decade ago that got my feet wet, but I’ve forgotten most of it by now.)

The single biggest thing I’d recommend is using the -tui option. I wish I’d noticed that one sooner.

I'm far from expert in linux debugging, but I liked my experience with VS code gdb support - you can set breakpoints, stepping in/out works fine, you of course get nice code navigation and so on. My experience was mostly with Postgres code base, which is pure c, FWIW.

Mostly because the ones that praise GDB know very little about the competition.

Early in my career I took a job at Boeing working on spacecraft simulation tools written in Ada. When I asked I was told by the other folks on the team that they had no ability to debug the system (i.e., they did debugging via print statements).

For background, our dev environments consisted of Windows PCs which we used to ssh (using putty, exceed, etc) into a SPARC/Solaris machine to do all of our actual work. Our tools consisted of vi (not vim), clearcase, and whatever other command line tools were available (good luck asking for anything new to be installed).

At some point, I guess somebody installed GNU tools on the solaris machine because I managed to find gdb while rooting around... which I'd never used before - so I set about learning. It proved to be a huge help. I tried to show it to some of the other engineers, but few seemed interested.

Unrelated but does anyone have tips on debugging async code in Nodejs. My work has a gigantic Nodejs monorepo and frankly I am struggling so pointers would be appreciated. There is dependency injection & layers of async / await abstraction.

Not fun, when I model my code / unit test against a working sample and some method 25+ layers down does not work as expected as it does not have expected input when the stack frame does not even give me the caller. Is there a better way than peppering the codebase with logging statements?

You can use the debugger from Chrome dev tools to debug Node.js. Just run node with the `—-inspect-brk` flag, open `about://inspect` in Chrome, and select your app under remote targets. This has saved me tons of debugging time. See also: https://nodejs.org/en/docs/guides/debugging-getting-started/

The GDB JIT interface implementation is seriously flawed. I love GDB - but this causes me a LOT of grief when debugging clasp (Common Lisp implemented using llvm as the backend https://github.com/clasp-developers/clasp.git).

Every time a JITted object file is added using the API, the entire symbol table is sorted. If you have 10,000+ JITted object files as we do - it takes hours to days to weeks to register them all.

We use the Undo time traveling debugger that builds on top of GDB. It's awesome but we are crippled because of the JIT API implementation.

I'd love to see this get fixed - if anyone knows who to talk with about it - drop me a line.

Have you peeked into it? What's the implementation like in there? Anything you could replace with an incremental merge instead of a full resort each time?

For my 98% of use cases these 3 gdb commands suffices : bt, f, p.

two more to add: b,l

On Emacs the GUD major mode, makes using GDB from the GUI quite convenient.

Does VIM has anything similar?

Yes. https://news.ycombinator.com/item?id=30760084

There is also vimspector, which is more advanced, a full blown debugging solution, using LSP and its lesser known cousin, the debugger adapter protocol. However, its quite hard to set up and success isn't guaranteed.

This article goes into a little more detail which may be useful: https://www.dannyadam.com/blog/2019/05/debugging-in-vim/

if you are using neovim, nvim-dap has support for debug adapters, which includes gdb in addition to several other languages (python, nodejs, java etc). Here [1] are available debug adapters.

It is noteable that nvim-dap just provides support for debug adapters, if you are looking for nice UI, there are other plugins such as nvim-dap-ui [2].

In vim land, there is vimspector. I have not used vimspector myself, but from what I hear, it is slower than nvim-dap.

[1] https://microsoft.github.io/debug-adapter-protocol/implement...

[2] https://github.com/rcarriga/nvim-dap-ui

Are there any easier to use C debuggers than GDB? All I really need to do is watch my variables change with each line.

You can use jetbrain clion an attach it to a binary. Or use even vscode.

If you're on Windows you can use RemedyBG: https://remedybg.itch.io/remedybg

A nice UI on top of GDB might be all you need. I've used this with some success in the past: https://www.gdbgui.com/

QtCreator has a very capable debug view. No need to have a Qt project, simply start a binary and step-in into it, or attach to a running process.

Most of IDEs have a similar capability to be able to use debug view without setting up a project.

Try GDB Dashboard, it makes gdb much easier to use:

https://github.com/cyrus-and/gdb-dashboard

There's also Voltron which works with both gdb and lldb (amongst others):

https://github.com/snare/voltron

Eclipse-cdt is a reasonable option for this.

Visual Studio, C++ Builder and XCode.

But if you really only need to watch variables any GUI frontend to gdb like QtCreator or DDD will do.

Visual Studio Code on Linux works really well and is increasingly popular, also a shout out for C-Lion. Eclipse is, umm, eclipsed by them both IMO.

My subjective opinion is that C-Lion is just a little bit more solid/complete, but Microsoft are investing heavily in VScode. It's great to see some serious competition by two well-resourced players working on this stuff.

Personally I'm always happiest with GDB tui. It's all the standard arguments of command-line vs GUI. When you know what you're doing CLI (with tab-complete) is just a faster way to accomplish the task, but when you're a noobie or an infrequent user the GUI wins because it's so much more discoverable.

How difficult is it to setup an editor like sublime text with gdb?

I'm curious if it can be an alternative to visual studio, meaning a window showing the stack, locals, watch, etc?

I don't know if it is possible in Sublime text (I never tried it), but there are several GUI frontend to GDB, including Nemiver and KDbg, as well as some IDEs allowing to use GDB to debug, like KDevelop and Eclipse.

I'd expect Emacs to have some GDB integration as well.

Emacs's GDB integration is probably my favourite (aside from using GDB directly):

https://www.gnu.org/software/emacs/manual/html_node/emacs/GD...

It gives you an IDE-like interface but supports easy access to the command line (as you'd expect, given Emacs's focus on text-based UX). Unlike (some of) the full IDEs it doesn't require you to be fully committed to a particular way of doing things (build systems, projects, etc).

I understand that VScode and CLion are also fairly lightweight IDEs and have GDB integration too, though ISTR VScode doesn't have watchpoint support yet. You can always type directly into the console, though.

It is possible, and it kinds of work but I didn't manage to get productive with it. It is far from Visual Studio or even QtCreator. I went back to the command line.

Emacs has GDB integration too, but it has been a long time since I used Emacs. Again, I don't remember being very successful with it and went back to the command line.

Edit: By Visual Studio, I mean the real one, not VS Code. VS Code is on the same level as Sublime Text, maybe a little bit better integrated. Among the different environments I have used, I consider Visual Studio to have the best C/C++ debugger, by far.

yeah, c with emacs and gdb works like a dream. Truly integrated deveopment experience. Scary langiage but great tooling. and valgrind works with them also, so your violating line pops up in the editor when it does a UMR or other memory error.

I used Geany editor (well, it's called a lightweight IDE) with a Debugger plugin. There's also another debug plugin for it, forgot the name, slightly more capable.

All of these are capable of the basic single-threaded debug operations. Multithreaded is not as robust, but it's quite a cookie per se.

For the basics the TUI is very much alright, as long as you remember the hotkeys.

Not entirely related, but the other day I wanted to add 6502 CPU (old stuff) support to GDB. The code of GDB is a nightmare to understand. I don't say it's good or bad, I really can't judge. But adding a CPU is scary, at best... Even if there is a supportive community and documentation.

Are there other debuggers out there that are easier to extend ?

Well, there's LLDB (https://lldb.llvm.org/) - I've heard it's got some nifty architectural features (e.g. having access to the Clang framework for handling C/C++ expressions).

I've done some minimal poking about in the code; I found its object-orientation a bit hard to grok (just for me personally) but it seemed to be quite uniformly applied so it might well be easier to work with.

I generally stick pretty close to the classic "old but still good" tools, but GDB is something I really could not get a handle on (probably because it isn't a daily-use tool for me).

Even if the rest of GDB remains a mystery to you, you can learn the two simplest and most useful functions: 1) how to create a backtrace, and 2) how to create a core dump. That way you can always send a bug report with lots of detail to somebody else for analysis.

> 2) how to create a core dump

This is a PITA under Linux, I figured this out but nor happy I had to look.

I wish Linux was like OpenBSD, where you get core files by default. This give me yet another reason to switch to OpenBSD, I only wish some other issues I am having with OpenBSD could get solved.

What did you figure out as a solution?

I've previously configured `/proc/sys/kernel/core_pattern` to generate core dumps by default (and not pipe them to a core dump collection program, which may be useful for error reporting but is rarely what I want).

There's also `gcore` for doing on-demand core dumps.

I wish it were possible to configure core dump behaviour per-process (or something similar).

This is what I did:

ulimit -S -c unlimited

sudo sysctl -w kernel.core_pattern=core

took me a while to figure this out, HTH

Thanks! I also see there's a `/proc/sys/kernel/core_uses_pid` that will make you get `core.PID` files, without having to mess around with core patterns - seems useful.

> OpenBSD, where you get core files by default

By this, are you referring to the behavior: "By default, this memory image is written to a file named programname.core in the working directory, provided the terminated process had write permission in the directory[...]" (from https://man.openbsd.org/core.5) ?

Also, do you know if there's a way to obtain a core file from a running process on OpenBSD? I don't see a port of gcore, and (e)gdb on OpenBSD doesn't support the generate-core-file command (it says, "Can't create a corefile").

Yes,

kill -ABRT pid

should do it

Good idea to just abort it. :)

However, I tried your suggestion from another terminal, and gdb appears to shield the program somehow. What ended up working is running "signal SIGABRT" from within gdb.

One thing that's still lacking is gcore's ability to take a snapshot of the core without killing the program. "Unlike after a crash, after gcore finishes its job the program remains running without any change." https://www.man7.org/linux/man-pages/man1/gcore.1.html

What do you use for debugging instead?

Print statements, haha.

My code is for linear algebra stuff, so it is pretty much a straight shot through/filling in some RCI loop most of the time, I usually just have to find which OpenMP or MPI barrier is misbehaving.

Print statements are really really good for debugging because they give you a timeline of "things that happened that I was interested in".

It's like using Google Maps to see the general shape of a landscape. A debugger is generally more like Google Streetview - you get strictly more detail and it makes some problems way easier to solve. It's just not always the easiest way to get an overview.

With that in mind, though, GDB's "Dynamic Printf" is pretty cool: https://sourceware.org/gdb/onlinedocs/gdb/Dynamic-Printf.htm...

You basically configure GDB to add printfs to the program you're debugging, without having to rebuild it. You just run under GDB and the prints will run. As a bonus, you can also interact using normal debugging commands too.

Never underestimate the power of print statements for debugging, that being said, the gdb-based "poor man's profiler" has also been a life saver.

Print statements indeed. That's when the source project is at your fingertips. Otherwise, it's more about collecting and figuring out the conflicting state, than locating the actual problem. So saving a core/logs is paramount.

Have not used gdb for a while but I think a good gdb tip is that you can evaluate functions with the print p command:

> p myfunction()

Getting more advanced, you can generally do this anywhere GDB will accept a value. Which means you can do things like:

> break my_file.c:123 if my_function() == 0x5

i.e. please stop at `my_file.c` line 123 but when `my_function()` is returning the value 0x5. GDB will handle calling the function and checking its value every time you get there.

Reply on news.ycombinator.com