GDB Debugging Full Example: Ncurses | Hacker News Reader

GDB Debugging Full Example: Ncurses

brendangregg.com

97 pointsokket21 comments

One of the coolest days I've had at work was when we hit a nasty deadlock deep in the bowels of gstreamer on one of our shared "immersion room" computers. We pinged our resident gstreamer expert and I convinced him to run his gdb session (from a core dump, over ssh, halfway around the world) inside tmux so we could watch him as he debugged.

There's nothing quite like watching someone who's an expert both in gdb and the problem domain as they do their thing.

normally on HN I don't like humble-bragging and therefore discount it. this comment is one of the few cases where I think it was done so well and massively worth it on the net. +1

"Immersion room" computers? What do you do for a living?

We develop interactive 3D applications at "room scale". http://www.oblong.com/solutions/

The top photo is of our immersion room staging environment. The versions deployed to production (at a handful of sites worldwide) have a little more polish :-)

Thanks, and sorry, this is totally off-topic... I am reading this as saying that there is some kind of data (or multiple kinds of data) which are worth viewing in multiple dimensions in giant holodeck-like rooms. I'm assuming remote conferencing is one of those, but I am interested in hearing more about the other kinds of data. Do you have some links to something a little more concrete than what's on the site you linked?

I just want to express how nice it is to see someone use vanilla tools on Linux to solve a problem. Whenever I see a thread on using gdb or jdb, invariably it is full of suggestions on the use of a half baked debugging wrapper that hasn't been maintained in 2+ years under the guise of it somehow being better because it has colors.

You can also launch the TUI after-the-fact using the key sequence 'C-x o'. (It is no accident that it is the key sequence bound to other-window by default in Emacs).

It's so surreal to see Brendan Gregg discussing Linux tools.

It's like watching Iron Man wielding Thor's hammer.

Which is not to compare Brendan to Iron Man, but it's as weird as that would be.

Thanks, but I should stress that my example here didn't do much that was really advanced. There was a little ftrace and BPF, gdb reverse stepping and the TUI, but the other gdb commands are routine. I guess what's useful is explaining everything, including caveats that you can hit even with the basic stuff.

No, I'm not complaining, it's just odd seeing you primarily work with solaris for so many years, and now talking about linux toolsets instead. I mean, I still remember you dissmissing SystemTap, although I believe you later said it had improved...

Speaking of which, as I understand it, dbx is to gdb as adb is to mdb. So other than syntax, what IS difference between the adb and dbx families, as you've clearly used both?

As far as reverse debugging goes, you really should look into http://rr-project.org/

I've been meaning to. I took a quick look now:

  # rr record date
  rr: Saving the execution of `date' to trace directory `/root/.local/share/rr/date-30'.
  [FATAL /build/rr-jR8ti5/rr-4.1.0/src/PerfCounters.cc:195:start_counter() errno: 2 'No such file or directory'] 
   -> Unable to open performance counter with 'perf_event_open'; are perf events enabled? Try 'perf record'.
  Aborted

oh-oh...

  # gdb `which rr`
  [...]
  (gdb) r record date
  Starting program: /usr/bin/rr record date
  [...]
  [FATAL /build/rr-jR8ti5/rr-4.1.0/src/PerfCounters.cc:195:start_counter() errno: 2 'No such file or directory'] 
   -> Unable to open performance counter with 'perf_event_open'; are perf events enabled? Try 'perf record'.
  Thread 1 "rr" received signal SIGABRT, Aborted.
  0x00007ffff6d6b418 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
  54	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
  (gdb) bt
  #0  0x00007ffff6d6b418 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
  #1  0x00007ffff6d6d01a in __GI_abort () at abort.c:89
  #2  0x00000000005a9133 in FatalOstream::~FatalOstream() ()
  #3  0x00000000006303fc in ?? ()
  #4  0x0000000000630679 in PerfCounters::reset(long) ()
  #5  0x00000000006cbd8f in Task::resume_execution(ResumeRequest, WaitRequest, TicksRequest, int) ()
  #6  0x0000000000638be9 in RecordSession::task_continue(Task*, RecordSession::StepState const&) ()
  #7  0x000000000063d70e in RecordSession::record_step() ()
  #8  0x00000000006354b6 in ?? ()
  #9  0x00000000006356f2 in RecordCommand::run(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) ()
  #10 0x000000000061f44d in main ()

I did a bit more gdb debugging (one debugger debugging another, although it's a bit hard as rr is stripped and doesn't have a debuginfo package!), and it looks like it's trying to use PMCs.

No PMCs here: EC2 Xen guest. So rr won't work, for me or EC2's 1+ million other customers. Perhaps rr can switch to software events?

There's a typo under heading 7.

...And thanks for the awesome write-up, makes something intimidating for a beginner like myself (gdb) seem somehow less so. Feeling inspired.

thanks, fixed!

I was trying to make it not intimidating by just writing what I knew, and _not_ researching all the odds and ends, instead, just saying "I don't know" etc as appropriate. Plus including my mistakes.

I'd like to add, as someone who is a novice programmer and trying to get more and more immersed in this world - it's posts like these that give you a small peek into thinking/being a programmer as opposed to doing programming as a job/profession/etc. Part of what makes programming and computers scary to beginners, speaking from my experience, is the notion that there is an infinite amount of complexity hiding just behind your IDE (things far beyond your grasp/ability to comprehend - i.e. You'll never know really how your program goes from the pretty syntax highlighted code in the IntelliJ window to x y or z app, or worse as in this case, when something goes wrong god forbid something with a sinister sounding name like 'segmentation fault'). Breaking that barrier down through the use of seemingly simple yet extremely powerful tools (Emacs, gdb, bash, etc) really helps, at least for me, feel less raw intimidation and self-loathing and more and more wonderment and inspiration to continue learning and hacking away.

Thanks for this. As someone who hasn't got much further than walking a stack trace and printing variables in gdb it's incredibly helpful.

Did everybody just discover the TUI in 2016 or something?

Not being flippant, just wondering because there've been a ton of links about it lately.

Probably, thanks to Greg Law's talk which featured here on HN, which I linked to in the post.

But this is often the case with Linux kernel and GNU tool features: they don't appear to have any marketing or sales professionals telling everyone about new features (unlike commercial offerings), which leaves us with many "hidden gems" (as people would say) in Linux + GNU. ftrace is another example.

It's been unsuable in the past. It segfaults gdb, the maintainers refuse patches and don't fix etc. If everyone is using it I guess the anointed committers got around to fixing all those null pointers..? Either way nice that something as important as gdb is actually being maintained for segfaults - assuming it is. I got burned by the segfaults and then the patch rejections so gave up.

> Did everybody just discover the TUI in 2016 or something?

Guilty as charged. I did just now.

But to be fair, I'd rather use a gdb frontend like realgud in Emacs.

Basically, yes. There was a well-watched talk about it that introduced it to people who hadn't seen it or hadn't bothered to try it.

Reply on news.ycombinator.com