The State of Linux Debuggers | Hacker News Reader

The State of Linux Debuggers

scattered-thoughts.net

237 pointspyb154 comments

I use vanilla gdb, the secret to the arrow-key problem the author had is they needed to switch window focus. The keybinding is Ctrl-X O, but if you don't want to remember that - and who could blame you - you can use the focus command, e.g. `focus cmd` will get you back to the command window, or `focus src` will get you back to the source window.

The display isn't really buggy, it's just the debugee output messed up the terminal. You can redraw it with ^L, or disable the tui while it's running. You can toggle it with Ctrl-X A, or just `tui enable` or `tui disable`.

I quite like the tui, the windows are configurable (use `show tui` and `show style`), and you can display more than just source, there's also registers and disassembly mode, e.g. `tui reg general`.

GDB is incredibly powerful but it really lacks some good tutorial, and some of its defaults are... questionable.

It's so heavily scriptable and configurable that you could argue that it's almost more of a debugger framework than just a debugger. I would argue that it's almost under-used really, simply because most devs don't realize what it can do besides setting a breakpoint and dumping a backtrace.

A cool thing that you can do with gdb is to change the values of variables in programs that are already running, or calling their functions. For example you can update a variable on your parent shell, e.g., "export" the variable $X to your parent (!) :

    gdb --batch-silent -ex "attach $PPID" -ex "call (int)bind_variable(\"X\",\"$X\",0)"

I confess that I use this trick in my shell scripts more than strictly necessary, just for fun.

That sounds both useful and horrifying. Do you use this for CI scripts? That's where I've needed this kind of functionality.

Never (yet) in production. I just use it for personal silly scripts, like changing the background color of my running xterms, and things like that.

Guilty as charged — I have never used GDB for anything but that, and I have never used awk for any other function than to select columns.

And you can make gdb send program output to a different terminal with the --tty option if you don't want program output to mess up your gdb windows.

Thanks! I know quite a few gdb tricks but I didn't know about this one, which seems quite useful if you need to debug an interactive console program.

> The display isn't really buggy, it's just the debugee output messed up the terminal.

That's clearly a horrible UX bug. The point of a debugger is to debug something which is probably going horribly wrong - having it mess up the debug output is a massive pain.

Clearly GDB could prevent this by intercepting the debuggee and preventing it from overwriting the rest of the screen.

Yes, just nobody has implemented it. It's kind of a pain. Also it brings up the question of whether GDB ought to implement a terminal so you can debug curses applications inside the TUI, which is even more of a pain.

Something something running GDB inside of GDB TUI…

> You can redraw it with ^L, or disable the tui while it's running.

I often run into a problem where switching to TUI after a crash disables local echo for some reason. I can't see my typing and TUI's output is totally busted. CTRL-L doesn't fix this, I have to exit the debugger and run "reset" in my terminal. Does anyone know why this happens or how to fix it without exiting the debugger?

I've even tried suspending with CTRL-Z, running "reset", then resuming gdb. This fixes the shell, but TUI is still busted, even with CTRL-L.

It’s a long shot, but try Ctrl+R. That re-enables local echo after it’s disabled in some scenarios (usually by Ctrl+S when connected to a remote session with the right tty settings).

I find close and reopen the TUI to be the only fix. A PITA for sure, but the only way I can get it to redraw correctly. Will try ctrl+l next time.

This is a great tip. My workaround to the arrow-keys issue was to learn the (readline?) shortcuts. ie. C-b/C-f for back/forward, C-e/C-a for end of line / start of line, etc.

Tavis, it would be great to see a screencast from you using gdb.

I don't think I have any of gdb, but I was just talking about a hobby project using frida! Frida is a tool that lets you inject v8 into any process, then exposes a javascript debugging api, I find it incredibly useful.

https://frida.re/docs/javascript-api/

https://twitter.com/taviso/status/1336693403510140929

Wow, that is very interesting. The docs say it is injecting QuickJS though, not v8?

> Frida’s core is written in C and injects QuickJS into the target processes, where your JS gets executed with full access to memory, hooking functions and even calling native functions inside the process.

https://frida.re/docs/home/

It supports both, and they've switched back and forth on what the default should be. They also supported duktape in the past.

I always use v8, because it has better compatibility with libraries and tools.

Might be a case of outdated documentation/website: https://frida.re/news/2019/05/15/frida-12-5-released/

Interesting, it looks like there was a move from V8 to QuickJS more recently than that: https://frida.re/news/2020/10/28/frida-14-0-released/

EDIT: and then the default moved back in https://frida.re/news/2020/12/01/frida-14-1-released/

Its been years since I needed it... but my favorite was always the DDD front-end (Data Display Debugger). Really nice X based front-end to GDB. The red hat TUI one wasn't bad either.

This is gold. Thank you.

Give it a try, but stick to vanilla gdb tui mode. It's worth learning.

I recently discovered a Linux debugger & tool which allowed me to solve problems 10x faster than gdb: pernos.co (which is layered over Mozilla's rr time-tracking debugger).

Pernosco's tool is described pretty well on their website (which is full of minimal fluff and marketing -- another good sign). But basically it allows you to view a program inside and out, forwards /and/ backwards, with zero replay lag. Everything from stack traces to variable displays (at any point in time in your code execution) is extremely easy to view and understand. The best part is the lightning fast search functionality (again: zero lag).

On top of this: extraordinary customer service if anything breaks (in my experience, they fix bugs within 24 hours and are highly communicative).

I'm not a paid Pernosco shill. I'm just so blown away by this product that I wanted to word-of-mouth spread it around because it's saved so much time for me.

That looks cool, but $20/month for 5 submissions is absurd. That's practically a product demo you have to pay for. Seems like the type of tool you'd only use on a really tough and rare bug, but I am not running into those consistently enough to justify a monthly payment. GDB's CLI might be ugly/annoying, but it's still usable and perfectly adequate.

Plus it's a cloud service I need to trust with my code? This is a very tough sell.

I might pay an annual fee for a local/self-hosted version though, kind of like a Jetbrains subscription.

You still have to email them to ask about for the on-premises version; they only just got set up to take credit-card payments for the subscription version. I happen to know that they don't have a sales department, so at least there's that.

As I understand it, it would be delivered as a pair of docker containers. One is for serving up the UI you see when you're debugging (I think there's one instance of this container per debugging session, but don't quote me on that), and the other ingests rr recordings into the database. A presentation they gave a while back stated that for the cloud version they send all recordings to 36-core AWS instances with 144 GB of ram, that it often takes as much as an hour to process a single recording, and that it can generate a 100GB database. Hopefully most recordings are simpler and quicker than that, but I don't think this is something I'll be running on my laptop in the next few years.

5 submissions a month means I can't just play around with it to figure out how it works. I'll have to learn its interface while also fighting some particularly difficult bug.

Maybe it's a great product, I'll never know.

Don't be so hasty, the first five hits^Hsubmissions are free.

Relevant recent(ish) HN discussion: https://news.ycombinator.com/item?id=25042827

> [Re: gdb -tui] Unfortunately, it's pretty buggy

I feel like this article gave up way too fast on gdb's tui mode, especially given that gdb works and tui didn't crash like half the debuggers in the list, other than "the command tui disable crashes gdb." which I can't reproduce.

The complaints about rendering and arrow keys are very easy to overlook with just a few tui mode hotkeys. ctrl-x-a toggles tui, ctrl-x-o toggles arrow-key focus between code and the gdb command line, and ctrl-l repaints whenever the rendering gets borked. Those three things cover all the concerns listed here.

The single most important thing about gdb, however, is the .gdbinit file. Being able to script your debug session is hands down the best thing about gdb. Bugs that require several steps to repro can be completely automated in gdb. You have conditionals and looping available in the debug script, this is very powerful. I suspect a bunch of the debuggers listed don't really support such a workflow.

Since I work in a CUDA environment often, it's also nice that cuda-gdb mirrors gdb, I can have scripting & tui mode for both.

> I feel like this article gave up way too fast on gdb's tui mode, especially given that gdb works and tui didn't crash like half the debuggers in the list, other than "the command tui disable crashes gdb." which I can't reproduce.

I've had the same conundrum for a various categories of development tools on Linux:

For a particular category (e.g., remote debugging, Vim C++ autocomplete system, etc.), an initial search uncovers 3-10 software combinations that people have documented as working.

And when I work through the list, trying to find one that works for me, I often hit problem after problem. It's hard to know if (a) that particular recipe is no longer workable, (b) there's something about my particular setup that will forever damn that approach, or (c) I just need to keep working through the problems until I finally get things working.

So it's hard to know how long to bang away at a particular recipe to determine if it's (a), (b), or (c).

The unfortunate result is I often run out of time for the investigation, and fall back to using Vim + gdb + ctags for developing on remote servers.

Mostly I'm just surprised that Visual Studio in 1999 gave me a better debugging experience than I've been able to cobble together on Linux in 2020.

I recommend Clion. It simply just works.

Good for you, but at work I’m not so lucky (large C++ project == CLion is so slow to be only usable if you’re a zen master).

For debugging, gdb -tui works for me (once I’ve learned to refresh the display after an output).

For coding, VSCode’s is much more responsive than CLion but its code navigation is inconsistent and it use >30GB of RAM once a week randomly (60 GB last week!).. I’m thinking about going back to Vim :-(

I've found clangd to be much more responsive than the built in LSP fwiw.

No comment on memory though, my remote dev env has 384 gb so I don't really pay much attention :/

Interested to know what is the size of the project? I'm currently working on something small about 60kloc of my code, another 60k in third_party and Qt5 as additional dep. There are a few hiccups every now and then when the Clion UI freezes for a few seconds but other than that it's been working really great.

It doesn't work very well for me - https://scattered-thoughts.net/writing/the-state-of-linux-de...

This is perhaps related to using zig, but it's frustrating that gdb itself works fine with zig but each gdb frontend manages to break in different ways.

Unfortunately ctrl-x-a also reproduces that crash for me - https://paste.sr.ht/~jamii/a1cd701eae0c2416eb9750d7c559e4b3b...

This was https://sourceware.org/bugzilla/show_bug.cgi?id=25348, fixed in June. I believe the fix is in the latest release.

If you can reproduce the crash, it's an ideal candidate to record with rr and try to debug :)

Oh I believe it crashes, but I’d speculate wildly this perhaps indicates some bad combo of gnu with other tools, or a missing lib, or a side effect of system customization, or a beta distro, or something like that maybe? Are you sure it’s an issue with gdb? I’ve used gdb a bunch, mostly on lots of Ubuntu machines, I haven’t seen tui crashing like that myself.

> some bad combo of gnu with other tools, or a missing lib, or a side effect of system customization, or a beta distro, or something like that maybe?

Oh, I'm sure it is. Debugging it would probably be good practice. I'll just open my working debugger...

The article should be named "the state of C/C++ debuggers and gdb in particular on Linux". Because there are debuggers for virtually any software ecosystem (for instance Pdb for Python) which frequently work the same cross platform.

Considering gdb/Linux: I love the Kdevelop GUI front-end for gdb. It allows introspection of variables in source code at mouse hovering.

Of course it also can open coredumps. The Fortran gdb is also great for debugging arrays.

If you don't mind OSS, check out https://totalview.io/ -- it is an instance of a domain specific debugger for C/C++ in HPC (massively parallel computing).

Eh, more particularly "the state of Zig debuggers and gdb in particular on NixOS Linux" - the author isn't wrong to want things to work, of course, but the author isn't using the most boring environment either.

This is true but note that:

* gdb works on zig code, so it isn't unreasonable to expect gdb frontends to also work * being on nixos might explain crashes or failing to start, but it's hard to imagine how it could be responsible for various gdb frontends not being able to set breakpoints, run to the correct line or render a pointer to an array of bytes, especially when those same features work fine in gdb * it also doesn't explain why the memory view in most frontends was buggy or didn't work at all * I also tried lldb, whose gui I couldn't operate at all, and code-lldb, whose breakage might actually be nix-related since I had to patch a binary (but the error message was complaining about invalid utf8...)

I'm also happy to try non-gdb/lldb debuggers if you have any suggestions.

> gdb works on zig code, so it isn't unreasonable to expect gdb frontends to also work

As they say typically in the license conditions: This software comes with absolutely no warranty.

Nothing that's not tested in every release (better in every CI build) can be assumed to work. How much of a standard boring Linux distro is written in zig? (That's an honest question, not a justification or even a defense to ship broken software.)

So yes, the headline is most likely misleading: It should be "The state of debugging zig binaries in NixOS." I expect that debugging a C binary on OpenSUSE[1] would give different results.

[1] I picked OpenSUSE here because installing library source is very easy and to my limited experience gdb just finds it. I haven't used OpenSUSE for a while. I have never used NixOS, so no clue whether that would rank above or below average.

Unfortunately, I don't really have a better suggestion. :-(

I know Solaris has its own debugger(s?), and rr looks really useful, but I haven't worked on a platform that could run either.

I use rr pretty often. Most gdb frontends also work using rr, as long as they either have buttons for reverse-step or allow you to enter commands directly. That's why I was kind of bummed out that gede, which was the most functional debugger I tried so far, has neither.

Most of the problems listed in this article seems to have nothing to do with Zig or NixOS. Why encourage finger pointing when you’re not sure about the exact reasons involved?

> Most of the problems listed in this article seems to have nothing to do with Zig or NixOS.

How are you able to make that judgement?

And if I go to that website it encourages me to "try it for free" which makes me think I am supposed to pay for it. But ok, let's give it a try. Then I need to enable a bunch of Javascript. Eh, ok. It was recommended on HN, probably not too malicious. And than it complains: Uh oh... we couldn't load this form.

The following may help you to view this form:

* Are you using an ad blocker? Try whitelisting this page or disabling it

* Browsing in a private or incognito window may prevent this form from loading correctly. Try viewing this page in a normal window.

* If you’re using Firefox Tracking Protection may be enabled. This could prevent this form from loading. Learn how to disable it.

* Try clearing your cache (including cookies).

Ok, close tab. I will keep using other tool for HPC debugging... (Yes, I actually do HPC and I of course have heard of totalview. Last I heard it is also a total pain to install.)

I remember TotalView releasing a product years and years ago called Memscape - Like Netscape, but for Memory. It looked like a browser, but was a sort of graphical heap profiler/leak detector.

I think it wasn't as powerful as Valgrind, but you've got to give them credit for an interesting UX concept.

It should definitely not be called that, since the author is not debugging C or C++.

I'm using zig, but there is also a fair amount of c in this project and I step into it fairly often. Gdb itself works just fine crossing between zig and c.

Yep, it's horrible. I use Vim 8's Termdebug. It just splits Vim in 3 windows - gdb console on top, program output in the middle, source code on the bottom. It does absolutely nothing fancy - you still type commands in the gdb window - but for me just having all 3 windows together is enough and it's super stable. In general any program that tries to do a fancy interface on Linux will be an absolutely miserable experience, there seems to be zero intersection between people who can write solid software and people who can design UI/UX in OSS and for some reason the two camps can't talk together. C'est la vie, I've just given up and use the console.

This is not really a criticism, just an observation. I suspect it stems from the fact that both software design and interface design are very complicated disciplines, but most OSS projects are done by a single person scratching an itch, so you either have programmers designing interface, or designers programming software, and both don't work out well. I suspect if we had some GUI libraries that were as easy as dumping text to console, the programmers would be able to churn out passable interfaces without having to do design. Quite possibly Tcl/Tk is that library, based on how rough, but usable gitk and git gui are.

"The State of Linux Debuggers". This is ... not that. Perhaps "One inexperienced user on an uncustomized system briefly trying several default-configuration front-ends (some of which are apparently malconfigured or missing dependencies) to GDB, without resorting to any documentation whatsoever, and gauging them solely on layout and visual appeal."

Pro-tip. When working with a new debugging environment "I haven't found a way to display the backtrace, list local variables or view memory" would be where most would look at documentation or seek advice, not move on to the next tool. That the author didn't is more of a comment on the reviewer than the product.

I feel this review is overly harsh. I felt the author was showing that many of the packages were not intuitive and were often buggy. He was showing me what I could expect when I might try them. When one expects one's users to RTFM, they've already lost.

> When one expects one's users to RTFM, they've already lost.

A debugger is a somewhat advanced tool, studying the documentation is absolutely required. It's a CNC machine, not a screwdriver.

(Agreed that gdb could use better documentation.)

That's possibly fair---I am by nature sometimes overly brusque. But I will say that debugging on Linux is hardly a neophyte's realm. Even properly using the debugger in purposefully less obtuse interfaces like Visual Studio is non-trivial and generally requires the assistance of walkthroughs, manuals, and colleagues/instructors.

Can't agree more. As a longtime user of Eclipse CDT on Ubuntu Desktop， I've not seen the kind of bugs mentioned in the article for more than five years.

On Emacs, I'd recommend using M-x gdb, rather than running gdb through GUD (which is M-x gud-gdb).

Emacs has a much tighter integration with gdb than with other debuggers. The features and interface are documented here: https://www.gnu.org/software/emacs/manual/html_node/emacs/GD...

After 10 years as a vim user, I switched to emacs in part for the gdb integration. (It turns out you can run emacs with vim keybindings and get the best of both worlds -- vi keybindings but a real programming language for scripting...)

You also get the sluggishness of Emacs, though. Slow startup and syntax coloring, compared to ViM and NeoViM.

> Slow startup

Simply run Emacs in server mode - it's an operating system, so don't shut it down too often ;-)

Worth trying Emacs 28. The speedup is impressive.

https://news.ycombinator.com/item?id=24117853

Daemon mode is your friend.

And it's much better when used with gdb-many-windows and speedbar

I prefer using dap-mode. It has a much nicer interface that's consistent across multiple debuggers. I use it for C, C++, Rust (Native Debug + gdb) and Python (debugpy).

this has been a total mess for me (cue the xkcd about the spacebar and thermal throttling).

using emacs in terminal mode its pretty frustrating to have the gdb interface bring up windows that you aren't allowed to delete...that and I can I really do more than 2 or 3. so whatever the intent was has been well and truly lost.

this kind of modality is anti-emacs...at least why my hindbrain thinks of as emacs

working on compiler-introduced introspection annotations and writing an elisp front end. we'll see.

What's even more depressing to me is that this is the experience that most students get when they first learn debugging. It's no wonder that so many professional developers don't bother using the debugger when they've had such an awful experience using it.

I just stick with ddd, QtCreator or KDevelop.

https://www.gnu.org/software/ddd/

https://doc.qt.io/qtcreator/creator-debugging.html

https://docs.kde.org/trunk5/en/extragear-kdevelop/kdevelop/d...

Back in the old XEmacs days, I used the gdb integration, no idea how far Emacs has improved on it.

I'm a big fan of cgdb;

https://cgdb.github.io/

Works great for native and cross compiled apps

That looks like magic. GDB should really add such functionality. It allready has dependency on curses.

Edit: Oh 'gdb -tui' ... I should read the docs more ofent.

There's also shortcuts to enter and leave tui mode:

https://sourceware.org/gdb/current/onlinedocs/gdb/TUI-Keys.h...

(I never start with gdb -tui but usually end up there for a little bit).

As the author pointed out, this hijacks arrow keys, so you have to use readline editing:

https://sourceware.org/gdb/current/onlinedocs/gdb/Readline-B...

https://sourceware.org/gdb/current/onlinedocs/gdb/Readline-M...

Fortunately, there's a lot of stuff that uses readline so it is all very portable.

I agree, cgdb is great. I've added some small things I want to my fork of cgdb for navigating windows etc.

I use gdb dashboard

https://github.com/cyrus-and/gdb-dashboard

It makes for a very pleasant debugging experience

https://i.imgur.com/iw21sbU.jpg

This is the best option I've tried so far. Thanks :)

Off topic but can you share your .tmux.conf? Thats hot af.

Sure: https://gist.github.com/jiva/954a22bafcafec2d594b5c3454a10b0...

Thanks a lot @jiva <3

I'm personally a big fan of vanilla GDB, at least for plain C debugging. Disclaimer: I spent >5years on a PhD work searching how to extend GDB for parallel and embedded programming.

    I can't easily see the surrounding code

I have Emacs on the side window, so I use GDB to see the immediately surrounding code, and Emacs to see the rest of the function. I do exactly the same for Python debbuging with PDB

    I have to manually request information rather than just glancing at the display

I just print the variables I want to see, there are not so many of them I'm actually interested in while debugging

    I have to remember syntax rather than just clicking on things
    ...and in this case, it took me a few tries to correctly deref this pointer to an fixed-size array

those are the tradeoff between flexibility and ease. You can access any memory locations, and cast it in many different ways, examine the assembly to precisely understand the next steps, examine the bits to precisely understand the storage (only endianness issues remained tricky to follow)

>"You can access any memory locations, and cast it in many different ways, examine the assembly to precisely understand the next steps"

This is not the exclusive domain of GDB. I have all those features in various Windows based IDEs all working in friendly manner.

> those are the tradeoff between flexibility and ease

It isn't a required tradeoff. Most of the gdb frontends I tried display the local variables at all time, but also allow writing arbitrary gdb expressions if I want to. Having both is clearly better.

It can't be a coincidence that literally every Linux C++ debugger majorly sucks. I think the blame has to lie ultimately with GDB itself and GDB/MI, because it's the only thing they all have in common (except for lldb I guess).

I think that the major problem is that gdb was never designed/suitable for graphical debugging. I use Visual Studio on C projects at work, and found it very interesting to find that Windbg actually has similar terrible usability as gdb. Gdb and Windbg were both command line interfaces originally, whereas Visual Studio has it's own debugger backend which was visual-first (afaik).

Really, VS is exceptional even in Windows land. I've tried RemedyDB, an attempted rewrite of debugging on windows, and found it useless. It had similar bugs to this article - no symbols, failure to attach, failed to bind breakpoints at all, failed to display variables.

I think it would be a huge project to replace GDB with a graphical-oriented debugger, but it would be worthwhile. If I were in charge, I'd make it a C lib that didn't come with a frontend. That way, the user can stay in their familiar editor and link the lib. (most programs should be able to use C libraries).

Totally agree that a debugger is the kind of software that majorly benefits from a GUI.

https://headcrab.rs/ is a project to create a modular debugger framework (implemented in Rust, but I think for debugging any native code) which could eventually form a good basis for such a tool.

Only if you use classical WinDgb instead of the new one.

https://docs.microsoft.com/en-us/windows-hardware/drivers/de...

Mac OS also is another example of having good graphical debuggers, even in the MPW days.

I disagree. I was using WinDbg downloaded from the Microsoft Store (so it had the WPF or whatever UI), and it was unusable. (Note: I was using it on a very large project - 4mLOC with hundreds of first- and third-party libs; It's likely I could get some use out of it on a trivial program, but it just didn't get the "real" program; and no, I wasn't simply missing the PDBs).

IMHO the original WinDbg UI was perfectly fine with the exception of a few small oddities; the new one is pointless trendchasing.

Well it is still preview, but definitely much better than the classical one, from my point of view, which looks like created by kernel developers first attempt at a GUI app.

it was in fact created using state-of-the-art windows gui frameworks. in 1995. and then never updated. is it really an improvement to have a win10 settings app that is all unified... except for most of the advanced settings and half of the basic settings which are only available in the control panel?

Win 16 nor Win32 were never state of the art Windows GUI frameworks, just the basic building blocks.

That place is reserved for OWL, VCL and MFC, which WinDbg doesn't use.

my mistake. I thought MDI was new in Windows 95, but in fact it dates back to Windows 3.0, and was in fact deprecated by 95.

I think WinDbg has much better commands than GDB, although that might be because I started with DOS DEBUG (of which WinDbg's commands have a strong resemblance to). Nonetheless there are some things in GDB whose design is definitely perplexing, like the "disassemble" command (https://stackoverflow.com/questions/39016138/how-to-force-gd... )

The only usable gdb-based debugger I have ever seen was Insight

But then it was killed because it was linking to gdb internals rather than using gdb's official api or something

> But then it was killed because it was linking to gdb internals rather than using gdb's official api or something

Oh? Their git repo[1] seems active.

[1]: https://sourceware.org/git/?p=insight.git

The last release seems to be from 2009: http://www.sourceware.org/insight/

I've always found bitter irony in that the buggiest applications are the debuggers. Just like the author of this article I've been annoyed at GDB's teletype oriented interface and wanted something better, only to find a lot of applications that are even more clumsy to use and crash more than the application I'm trying to fix.

This goes back a long way too. xxgdb used to be reasonably stable but these days it seems to crash quite frequently. ddd seemed so awesome when I first discovered it, but it too failed spectacularly on the various occasions when I tried actually using it. I occasionally go back to see if they've fixed the bugs but thus far have not had any luck.

How about this reddit comment? https://www.reddit.com/r/emacs/comments/6p872z/make_emacs_pa...

Here it is for those who don't like reddit

========================

I think this approach by the programming profession towards tools that are critical to their occupation is somewhat ass-backwards.

Their idea of professionals depending on critical tools which are created by other professionals in their spare time is wrong. It is time software engineers created guilds that fund the development of tools they need and use. It may be that software development is a new profession which is failing to get properly organized on account of the involvement of big corporations. But that is a throwback to an era when computers were expensive resources only big companies (with lashings of DARPA and NSF funding) could afford and thus were able to influence the tools and the outlook.

Those days are long gone in an era where a powerful computer can be had for less than $200, and there is no need for professionals to allow corporate agendas to undermine the proper development of software tools, as it has been since the internet came around, and it is time software developers learned to see themselves as professionals and stop seeing their jobs as gigs as though they are part time semi-employed musicians or something.

I mean it is 2017 and GCC has been available since the 1990s, but where is the graphical C++ debugger rivaling Visual Studio that we would have by now if developers belonged to a well organized profession which saw the need to fund the development of their software tools through membership dues just like any of the professional bodies out there? By that I mean the software developers themselves, wholly independent of any corporations out there, ie no Linux Foundation type f&ckery where BigCo come in and shape the direction to their agenda.

The software could be Free GNU License software but the hard part would be done by paid professionals funded by guild membership fees.

It's an interesting idea, but like it or not, companies DO set the standards for corporate software development, and they unfortunately don't understand the value of wasted time. A company would rather tell their devs to use free tools like GCC than pay $x,xxx per head to give them something world-class like Visual Studio.

Visual Studio still has the best the debugger. It’s not perfect or even close. But it’s 50x better than cmdline gdb.

RemedyBG is a cool one-man side-project debugger. It’s crazy impressive for what it is. However it has never worked for me on any professional project. There’s too many little one-off issues that are hard blockers. I wish it was open source. https://remedybg.handmade.network/

Not exactly the same as other debuggers, but the "rr" debugger could be given a look.

Highly recommend using rr if you're in an environment that supports it. Once you've learned to use it, it feels like a debugging superpower. You can use most of the gdb commands you are familiar with, and most of the gdb interfaces listed here will still work.

The docs are here: https://rr-project.org/

Have you tried Pernosco yet? It's a meta-superpower; I use it almost exclusively now. https://pernos.co/

No, that looks really slick, though.

rr is nice when it works, but I've only had it work on a single system of mine (recent Intel). gdb is portable and works most anywhere. If you're looking at other architectures, you'll almost certainly have a gdb port but probably won't have a rr port.

It looks like some changes have been made recently:

- https://github.com/rr-debugger/rr/issues/1373

- https://github.com/rr-debugger/rr/issues/2034

Really modern AMD processors will work in rr. AArch64 support is pretty experimental still. But any Core or Xenon Intel processor you bought in the last 8 or 9 years should work.

Also if you have trouble please file issues!

I wonder what results the author would have had if they'd tried to debug a binary written in C or C++.

Yeah I think that's really the problem. Also, they're using Nix.

I haven't had problems getting source locations to work with gdb console, Eclipse, or CLion on Ubuntu 16.04, debugging C++ binaries I built with a shell script.

Try a new debugger tui that I just built a few months ago: Blinkenlights: https://justine.storage.googleapis.com/blinkenlights/index.h...

Looks interesting. I'll add it to my list for the followup post.

I've used ddd for debugging statically compiled code, C C++ and Fortran. It is quirky. Its weirder with MPI, as you get lots of debug windows.

For dynamic languages, Perl Python and others, I use komodo from ActiveState. Been a customer of theirs since 2007 or so. Excellent tool, though with a few quirks as well.

Julia has/had a native debugger, and an atom plugin, but seems to be moving towards vscode. Vscode is many things, and is highly opinionated. I've gotten the debugger to "work", albeit not being helpful for the Julia I was working on. I tried the C/Fortran plugins for it and was disappointed.

I wasn't aware that totalview went OSS. I thought it was closed.

I think you would love pwndbg

https://github.com/pwndbg/pwndbg

I'll second pwndbg, but also recommend gef. Gef is a little more lightweight and optimized. It doesn't include the reverse engineering tools that pwndbg has, but it doesn't sound like you're using those anyway. For general debugging it should work the exact same way as pwndbg.

https://github.com/hugsy/gef

I keep going back to Vim + commandline gdb. However, QtCreator + CMake on a large project worked well enough to be productive.

One I haven't seen anyone mention yet but also haven't tried myself is https://codeclap.handmade.network/

It's not free, but it looks like a high-quality cross-platform gdb and lldb frontend with an imgui UI.

I'll add it to my list for a followup post.

Have you tried https://github.com/cyrus-and/gdb-dashboard ?

Edit: just saw that several people mentioned this already. Good, it is a nice tool!

The biggest issue with debugging on Linux is still the awful state of debug symbols. Linux famously lacks the equivalent of Microsoft’s symbol server and that changes almost everything to the worse.

That's not true anymore, but it's not widespread yet. https://developers.redhat.com/blog/2019/10/14/introducing-de...

Agreed. Fedora/Centos/RHEL have debug packages though, so the symbols are available for most software that you've got on your machine. GDB prints out a message reminding you to install them, but doesn't actually do it for you. We also have PolicyKit, so in principle GDB could just ask for those packages to be installed. We should close the gap, and while we're at it we should make PolicyKit not ask for the root password when installing debug packages. Then it could be a seamless process, at least for software installed from the official packages.

Symbol server is critical. I wish the whole debugging environment was even simpler though.

Binaries. Debug Symbols. Source Code.

I wish those three things could be trivially merged and split. For internal builds I want to deploy a single binary that contains symbols and source code. This way issues can be trivially debugged by anyone on any machine. For public builds I wish to split out symbols and source which can be hosted on a symbol server.

Anyone have recommendations of graphical "reverse engineering debuggers"/binary-only debuggers (not sure about the name of this genre) like OllyDbg or x64dbg for Linux?

Try radare2. There is also GUI fronted called cutter (https://cutter.re/).

Evan’s debugger gets you most of the way there: https://github.com/eteran/edb-debugger

You have to build from source but it’s easy to do.

I’m thinking of devoting a summer to writing an Ollydbg-like UI wrapper for LLDB, because I miss Olly so bad when I leave Windows.

Not sure if I understood what genre you have in mind. Does Ghidra fit the bill?

I meant a debugger that doesn't require or expect you to have the source code of the application you're debugging and just shows you disassembly, lets you use instruction execution/memory access breakpoints, patch instructions, etc.

Ghidra doesn't have a debugger yet, but it's a planned feature.

Ghidra doesn't have a debugger (which afaik is a WIP).

It is bad with C/C++

Unfortunately, I use Pascal and there it is even worse

GDB tries to parse Pascal syntax in print, but many expressions do not work. It is almost a mix of Pascal and C.

The integration in the Lazarus IDE makes it even worse, when you cannot enter all expressions. And there are so many bugs. You get GDB crashing, Lazarus crashing, or GDB freezing the entire X server if it pauses the program in a bad moment. Then you need to restart the system or kill GDB from outside the X server.

No-one maintains the Pascal support in GDB. In fact, I think the test suite doesn't even run any more (at least for me, FPC seems totally broken). This is an area where a volunteer could make useful progress pretty easily.

On linux it works fine, on windows it sucks but there are some workarounds to make it usable : https://wiki.freepascal.org/Debugger_Setup

This is a great resource, hadn't heard of most of them. I still think Eclipse is fantastic although not something for a quickie session, for that I'd probably use command line or the simple curses GUI.

I have to say, though, much as I enjoy working on and with Linux full-blown Visual Studio Enterprise is really in a league of its own but not really complaining that a similar debugger for C++ under Linux doesn't exist yet.

The rendering problems in curses programs could easily be the fault of his terminal emulator or even an incorrect value in his TERM variable.

Is there an easy way to check if everything is set up properly and also to get tips on what needs to be changed?

Not really, but the most common mistake is to set TERM in your shell startup files (.bashrc, for example). If you do that, then you could end up setting it to a value not supported by your terminal emulator. If you don't set it, then your terminal emulator will set it to an appropriate value for you.

I thought one needed Ctrl+L to refresh the screen when it gets corrupted by program output in "gdb -tui".

That's absolutely true, I'd forgotten about that. It's hard to tell from the screenshot if that's what happened, but it's quite likely.

What about https://en.wikipedia.org/wiki/Strace or Geohot's timeless debugger Qira https://github.com/geohot/qira

strace isn’t really a debugger per se.

I use `gdb-dashboard` along with others, but mine is heavily modified. I wrote about mine at https://gavinhoward.com/2020/12/my-development-environment-a... .

I'm currently using pwndbg, which is really just a few python enhancements to the gdb cli, mainly focused on developing exploits, hence the name. It's good enough for me, for now.

Something I've realized about gdb in general is it gets a lot better once you figure out the python api and use it to ease your repetitive tasks.

> Eclipse does not play well with my window manager.

For sway, _JAVA_AWT_WM_NONREPARENTING=1 solved most of my issues with java apps.

Nothing about the proprietary options, green hills etc. I recall green hills had reverse debugging about 15 years ago, 5+ before gdb. I never used it in anger.

Anyone got a link to anything that compares things that aren't just gdb frontends or under-developed workalikes?

I've been avoiding proprietary options because it's usually a big time investment to get them working on nixos, but I'm open to suggestions.

I quite like gdb dashboard.

https://github.com/cyrus-and/gdb-dashboard

The state of Jamie Blogger

> The gui doesn't seem to have a way to open an executable.

Activate the "Load Executable" button in the toolbar.

> Using "Open Source Code" produces a file dialog that doesn't seem to have a way to paste paths.

Paste in the location bar or in the file name field, or access the context menu of the location bar and activate the "Paste" menu entry.

> Passing the test exe at the command line (kdbg ./test) open up an editor at start.S but still doesn't seem to have a way to run the exe.

Activate the "Run" button in the toolbar.

----

Verdict: too fucking rushed that he doesn't use common sense and tries the obvious things.

> Activate the "Load Executable" button in the toolbar.

There is not a "Load Executable" button in my toolbar.

https://imgur.com/a/qOTZgy3

I also mentioned that in the article. Maybe you read it in a rush.

> Paste in the location bar or in the file name field, or access the context menu of the location bar and activate the "Paste" menu entry.

The location bar does not accept text. It's just a dropdown. Pressing C-l doesn't replace it either, like it does in the gtk file selector.

https://imgur.com/a/wbuKniU

> Activate the "Run" button in the toolbar.

There is no "Run" button in my toolbar.

https://imgur.com/a/rUHMRam

https://i.imgur.com/9Y0JXkO.png

https://www.kdbg.org/kdbgtotal.png

There's also the paid VisualGDB and you could probably make Visual Studio work for debugging a remote application via SSH.

Evan's debugger (edb) is pretty good too. The Linux equivalent of OllyDbg.

> The text boxes for eg the memory window ... didn't consistently accept keypresses at all

Old Motif/X apps follow "focus on mouse hover" principle, so you can modify text fields only when there is a mouse cursor on them.

I can't believe this comment ended up dead - that was the correct answer. I had more success with ddd this time around.

I guess, writing debuggers is hard and if you need to support multiple environments or console debugging it's probably wise, as a programming language community, to simply make your users adopt print-statement- or gdb-debugging.

But as a long-time Visual Studio developer (C# and C++) I am constantly amazed how bad that status quo is for other languages. I'm probably spoiled by all its features:

* Conditional Breakpoints.

* Move instruction pointer.

* Jump between threads, freeze and thaw them.

* Showing variable Values when hovering over them.

* Inspecting (large) Lists and Dictionaries easily.

* Changing variable values or execute methods while breaking.

* Changing code and continue debugging.

* Historical debugging (moving backward)

I believe that someone who never experience debugging like this cannot possibly imagine how much you could improve debugging in other environments.

I think those are all basic features that all major debuggers support, certainly gdb can do those including moving backwards (gdb calls it reverse debugging).

- Conditional breakpoints, break main.c:26 if a > 6

- Move instruction pointer, jump main.c:84

- Jump between threads, thread 123

- etc, etc.

I've barely used Visual Studio, but I'm very familiar with windbg which uses the same debugger engine with a different UI.

Reply on news.ycombinator.com