What a good debugger can do
werat.dev
werat.dev
I like many of the features that debuggers can provide, and I think this is a good article to set aspirational goals for what is possible. But my lived experience has generally been that it's a buggy, fuzzy, moving target in terms of overall user experience. GDB, bless its heart, is still somewhat a pile-of-bugs itself. I frequently have it crash, or otherwise get into a "so confused it needs to be restarted" state. Visual Studio is much more stable, though less powerful, too — less scriptable, at least.
Perhaps some day, debugger tech^H^H^H^H UX will advance to the point where it really delivers on its promises consistently and solidly. But after 20+ years in software, I am not holding my breath (: There are some situations where a debugger is just the thing you want (eg: hardware breakpoints can be a life-saver), but I find that's more the exception than the rule, at least in the corners of the software world I've worked in.
Compare the above with logging: It is simple and trustworthy. As you get good at designing a solid logging system, and interpreting the results, your life just gets better and better. If you get good at using a debugger, you can still be hit with gnarly weird behaviors, debugger apoplexy, optimized-out-code wackiness, etc. that are hard to control or predict.
Anyway, "debugger vs. logging" are often presented as some sort of either/or choice, and in some sense it is (you only have X time to spend; where would you like to spend it?), but in many senses it is not; both have their strengths. I just find that the cost/benefit for me has generally favored logging and testing, over the years.
Sometimes you can improve the situation by sticking things in a bigger memory buffer and tightening control of when things get flushed to disk, but there's always that fundamental "observing the system will change it" problem. Similar issues arise with a debugger too, of course.
This sounds so naive for someone with 20+ years in the field...
Linux, for decades, couldn't get logging to the point that it at least doesn't lose messages (the problem with tail / logrotate that is quite obvious once you think about it, but it took many years to give up the approach).
I recently hit a bug where NVidia's driver abuses Linux kernel logging in some tight loop by spamming log messages at insane speed (happens when you have two video adapters, Intel and NVidia and an external monitor). An interesting side-effect here is that Linux logging tries to throttle loggers who output too much, so, from the log you cannot tell what's happening (because even though the system is burning calories trying to print a tonne of messages, nothing really gets printed).
Several iterations ago I worked on a product where logging had to be implemented as writes to shared memory self-styled circular buffer, and because there was too much info printed too quickly you only had few seconds worth of logs before system crash... on a good day.
Needless to mention the fun of stitching together logs coming from different places in your system with separate clocks.
Even simply processing hundreds of Gigabytes of logs on its own isn't a trivial task.
----
Many things are simple, when your task is simple. Logging is just one of those things.
I agree with much of what you said, and of course "logging" is not just a single point in the solution space — there is some function "troubleshooting_pain = f(your_project, your_approach)". I was trying to say that for "your_approach=logging" that function tends to return smaller values than for "your_approach=debugging", all other things being equal, in my experience.
Whereas your comments seem more oriented towards the "your_project" factor. Of course using logs is harder on a distributed system. But so is using a debugger, or just about anything else.
Perhaps I should have said "It is relatively simple and trustworthy, even if it can still get hairy at the extremes."
Things like EBPF (which may implement sort of a declarative debugger) are, perhaps the only tool you may hope to use in high volume and high frequency systems.
If I could only choose one technology used for software diagnostics, I'd choose debuggers over logging. Debuggers need more effort to develop them, and they aren't very good (yet), but they have potential. I don't believe that logging can be substantially improved to deal with difficult problems.
I guess my (limited) experiences with distributed systems are different than yours. The notion of "pausing" the system to step through things interactively was usually untenable. Do you stop the one node that shows the issue, and let the others run, getting into who-knows-what shared state? Do you somehow attempt to stop them all, and hope/pray that they all are in the right state to make your cross-node analysis meaningful? This was mostly on Apache Spark, where parallelism was the name of the game. Maybe for some kind of long-running distributed system like Erlang it's a different story.
Not just that :) It was "inspired by". Well, it's the same idea.
> The notion of "pausing" the system to step through things interactively was usually untenable.
That's not what EBPF would be used for in such a system. You'd write a bit of code that can be loaded into a running program and executed as a particular condition occurs. Like how you can attach some code to evaluate on a breakpoint in many other debuggers.
...
Then, you may notice that TUI mode steals certain keyboard commands, such as up/down arrow to scroll the source listing, rather than navigating command history.
Then, you might enable Vi-mode for GDB's readline, so you can use "j/k" to navigate command history, even in TUI mode. Plus Vi-mode is just better (:
Then, you may find that certain things don't work quite right in Vi-mode, because it's not the default and doesn't get as much testing. But you fuddle along because it's better than the alternative.
And thus you have arrived at my basic situation (:
For Javascript, I use Brave/Chrome debugger.
...And I should use logs more.
I am always in shock when I see people work on large projects with no debugger. I do printf/console.log all the time but HOW DO PEOPLE NOT STEP-STEP-STEP??
I think I have felt some of what you are feeling. The debugger is ... seductive (: For one thing, it is an educational experience, as you step along you are learning what your program is actually doing, which is a good mental checkpoint, especially when I was a more junior developer. I think a debugger as an educational tool is a big point in its favor. And so you may develop a positive relationship with your debugger, and want to bring it with you wherever you go.
But you may find, as I did, that the sparkle of that relationship fades somewhat in time. You will be better at knowing what the code does, without needing the debugger to tell you. You will hit situations where the debugger cannot help you, or is less helpful than the alternatives. You will get better at structuring your code so certain classes of problems simply don't happen as much, rather than using a debugger to peek-and-poke to fix things all the time. You may find the debugger to be a relatively exhausting high-touch real-time experience, compared to thinking about issues more at your leisure, or getting a larger understanding from other sources.
And so, with experience, and hopefully with an open mind, the debugger will settle into its proper station, in the pantheon of your various debugging aids. Not bad, but not the end-all-be-all either.
I am being somewhat flowery in my language, but all I am saying is: it's okay to rely on the debugger, at some stages. Give it time. Just don't forget about the alternatives or denigrate them needlessly. You want to build a big happy family of technologies and techniques (many of them residing solely in your mind) that can all work together.
For me it’s because that is slow, tedious and exhausting. Often you need to cover a lot of code surface area to find a bug.
Worse, the bug may be something reported in production in a giant pile of code.
A good set of logs will go a very long way to at least narrowing down where the bad behavior is coming from, and is generally scalabale.
The debugger is useful, but generally only at the extremes of “new to the code, totally clueless” and “really advanced bug that I need to bring out the big guns for”.
The logging approach clicks in really fast the minute you’re talking about a multiprocess program, being microservices or just plain old SOA.
When you have a bug that doesn't happen until hours into your program execution, being able to set a breakpoint, edit-and-continue, and set-next-statement are worth their weight in gold. You can solve in the moment what would take literally days, even weeks using a logging approach.
Sure they both have their place but having a debugger is indispensable, and Visual Studio's debugger is the undisputed king.
Well, I personally do agree it's a better, more solid overall product. And also agreed that edit-and-continue in some highly stateful situation can be ::chef's kiss::
But I think there is room to dispute its kingliness (:
For instance, scripting GDB with Python is quite nice, on occasion.
Actually, just on Windows, WinDBG has some killer-feature functionality of its own.
Or back with GDB, take, for example, this classic post by Jonathan Blow, where there is some good back-n-forth discussion about GDB vs VStudio, with varying opinions:
* https://news.ycombinator.com/item?id=5125078
Plenty of disputers out there, I daresay.
Breaks on enough codebases for me to not be in the habit of relying on it.
My own "killer feature" of VS debugging is being able to open up a crash dump, have VS auto-download matching pdbs from a symbol server, and auto-view the correct source file version/revision thanks to source indexing - without checking out by hand and turning my build stale. Killer ergonomics.
Sometimes I'll resort to windbg/cdb for memory pattern searches, automation, untangling wow-mangled crashdumps, etc. but VS itself is a nice first resort.
Is it though? Add concurrency and it's no longer simple. Have an operation running millions of times a second? Good luck even logging that fast enough. If you start adding logic to print only sometimes, now you get a poor-man's debugger.
Yet for me, in those high-concurrency/high-frequency situations, I feel like a debugger is also a pain to use and trust? In truth, I've had best luck solving those sorts of issues by old-fashioned thinking through it / talking it over with someone / creating hypotheses and testing them sort of analysis. And even for that, logging (or profiling, event tracing, and similar "dump lots of data" approaches) tends to produce better "see the big picture" information than a debugger's laser-like focus. Might come down to one's personality somewhat, too (:
* Relatively early in your project, incorporate a "real" logging system, and start leaning on it. For me, in C++, I have most recently been using 'spdlog'[1] fairly happily. It accumulates data in memory and dumps it from a dedicated logging thread at a configurable frequency. That approach helps avoid logging getting in the way of your main program performance, but it has downsides (see below).
** You want logging to be easy to add wherever you need it. It should be about as easy as adding a literal "printf" statement; minimize the barrier to making use of it.
* Be able to slice/dice your logs by category of information, not just debug/warning/error. I find myself often making special "SYSTEM_XYZ_LOG(...)" macros, for different systems. Then, with a compiler flag I can enable/disable output for different facets of the program. Some of those I might leave on, and others are so specialized (or performance-impacting) that I only enable them when needed.
* It can be nice to have several log files, one per system (eg, in a video game: graphics messages go to one file, physics to another, UI to another, etc). However, it can also be useful to have a "combined" log that shows everything intermingled, so you can see the overall timing of things without having to correlate timestamps across files. Ideally your logging systems supports having both.
* Along with the above, develop an ergonomic way of viewing various logs at once. My approach is pretty simplistic: I have a bunch of XTerm windows, and some of them are displaying logs, sometimes filtered variously (eg: `tail -F foo.log | grep -C5 interesting-pattern`). You could get fancier with tmux or something.
** Aside: the general skill of being able to wrangle lots of data from various text files is a good one to develop — not just for logging. I find this much easier to do in a Unix-style environment than on Windows.
* There is a tradeoff between logging synchronously (eg: writing to STDERR, unbuffered) or accumulate-then-flush (such as a buffered STDOUT, or a logger thread approach like in spdlog). If your program crashes, you might not see the most recent (and thus most relevant!) log messages in the latter approach. I usually have some "write to stderr right friggin' now" function for special cases when I need it. However, if you run your program in a debugger and the crash happens, you might be able to step the debugger to let the logging thread dump out whatever is not-yet-flushed-to-disk. I have had good success with that; I just have to remember to do it.
* When you generate too many logging messages, there is a tradeoff between "flush to disk to free up memory" and "just allocate more memory". If you are generating huge amounts of logs, it is possible you will use too much memory on the system, in the latter approach. But in some cases, the latter approach is faster. I've had good enough luck using the former approach and just using a "plenty big" memory buffer.
* If you are logging across multiple machines, you usually need to correlate events via timestamps. So, make sure your clocks are synchronized, or at the very least be aware of this issue (can be a pain).
* Don't be afraid to allow big sizes for your log files, at least when debugging. Storage is cheap, grep is fast. Depends on the scope of your project, of course.
* There's a difference between the logging for your codebase in general, and logging for very specific debugging purposes. I find it handy to have a special log for the latter case, which generally is empty, but when I am investigating something, I can write to that log and monitor it specially, to cut down on having to sift through a lot of noise. It is just the "debugging the current problem" log, and I delete those log statements once I am done. This is essentially the same as using ad-hoc "printf" statements, but using the real logging system, with the benefits that affords.
* Ideally, your log system should not construct its message unless it is actually wanted. Eg: if your log level is "warnings or worse", then LOG_INFO("foo={}", someValue) should not perform any string building work. This seems fairly common today, but some logging APIs don't get this right.
* In C/C++, logging should go through a macro, so it can be compiled out (or compiled out beyond a given severity level) depending on your build. spdlog supports this, and it is fairly easy to write your own, typically.
* A nice-to-have feature is to be able to only log the first N of a duplicate message, when desired. Sometimes (especially when things go wrong), your program will produce an outrageous amount of the same log message, which just drowns out the useful information and potentially use lots of memory. Some logging systems have explicit support for this concept. You can also roll your own (eg: by adding a timer or counter guarding some logging).
* Another nice-to-have feature is having a notion of pushing/popping scopes for logging (eg: log4j's "NDC" concept). In your code, this would correspond to lexical scopes. In the log statements, it would come across as some sort of "toplevel>outer>inner>" prefix or so. This is one thing I wish spdlog had.
** Aha! In digging up the docs for NDC, I found this[2], which does mention a book for your reading list: "Patterns for Logging Diagnostic Messages" part of the book "Pattern Languages of Program Design 3" edited by Martin et al. I cannot vouch for it.
And, as mentioned variously in this thread, logging is just one tool in the toolbox. Don't forget about performance counters, even traces, writing good tests, the debugger, etc. Good logging requires some up-front cost, but generally worth it, IME.
[1] https://github.com/gabime/spdlog
[2] https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4...
Always enjoy reading people enthusing about tooling they know well.
Git bisect to the rescue, what do you know, it breaks on the commit adding another improvement to the logging system. The second message is getting eaten by the logging system, that's all.
I don't think this persuaded the co-worker to stick to the more common tools, but definitely fortified me in my belief that a buggy tool you made yourself is no way better than a buggy tool that thousands use all the time.
a good debugger supports different kinds of breakpoints, offers rich data visualization capabilities, has a REPL for executing expressions, can show the dependencies between threads and control their execution, can pick up changes in the source code and apply them without restarting the program, can step through the code backward and rewind the program state to any point in history, and can even record the entire program execution and visualize control flow and data flow history.
I should mention that the perfect debugger doesn’t exist.
https://cuis-smalltalk.github.io/TheCuisBook/The-Debugger.ht...
Ideally all debug features (breakpoints, watchpoints, tracepoints &c.) will work running both forwards and backwards. To meet the standard of the "perfect debugger" from the TFA intro, I would say you should be able to run backwards, modify a function, and run forwards again. I'm not aware of any debuggers that let you do this.
Executing arbitrary code in the program's context is also a regular debug feature, but I don't think many reversible debuggers allow this.
Pernosco supports this too.
I remember once debugging a thread without killing it. I've asked a customer to click on a link that would get to a bug while having remotely opened the IDE that was serving that user session, setting a conditional halt. Seen it halting, fixing the bug and saving the method with the halt removed and let the thread run. All the user saw was a long request that ended in service instead of that bug.
"The stack of (framed) execution contexts gives a history of the computation so far. You can select any frame, view instance values in the receiver, view the arguments and method variables at that point."
So step backwards, modify a method, step backwards before that method send, and resume execution with the modified method. (Note: resume rather than restart, so the modified method has the preserved context unless we manually edit that context.)
This seems impossible for anything that modifies external state.
Say 1. Open database connection 2. Commit data 3. Close database connection
How would you rewind right before #2 if you've already completed step #3? You'd need the socket/connection you already closed
Unless that means something more like "keep a running record of program state over time"
Here's an example: http://docs.mech-lang.org/#/examples/bouncing-balls.mec
If you want these kinds of features in other systems, they'll have to be architected to support them. For example, the external database will have to be rewound as well. If it doesn't support that feature, then cool language-level debugging features won't be as useful.
One use case I have is for instance debugging a bug that's hard to trigger. When it finally happens and my breakpoint is hit, I can edit the code, hot swap it live, drop the current frame and then make it call my updated function again as if the original call never happened.
It works on recorded state. It's not about executing a program again, it's about root causing a failure by going back in time after the failure happened. You can do things like start with a crash, retroactively set a break point and reverse run back in time until you hit it.
It's for real world applications. It was written specifically to debug Firefox, and has since been used for other applications of similar size.
It's basically GDB with extra commands, so very easy to use and learn if you know GDB. Highly recommend.
[1] Necessary, but not sufficient. Multithreaded applications require a lot more care, and rr relies on very accurate hardware counters to get multithreaded executions working correctly (and not all hardware supports these hardware counters!).
The key thing is that you're really rewinding the world, as visible by the program
The program doesn't actually know what the network transaction with the database was, it just knows what syscalls it made and what results they returned. If, whenever it gets to talking with the database, you provide the same result as last time then it can't tell the database is gone.
This is self-supporting: if you do this consistently with external sources of information then the program will never go down any new code paths, guaranteeing that you'll always have a ready answer recorded when it needs it.
You would need a lot of storage space for some programs, but in simple cases that might still be useful.
So, in your example, playing the program back wouldn't really try to read from a closed socket, it would just hit debugger's database of stored system calls at a particular point and retrieve the stored response from that call.
This could get weird though if the program modifies itself as it executes. Not sure what to do in such case, but maybe there's a way to deal with it in special cases, not in general...
I am a co-founder of undo.io, many of our customers do this.
It's not as bad as it first sounds because the replay of the program will modify itself deterministically. (Though as always with this stuff, there are some gotchas.)
You can rewind & replay a fixed execution of the program.
All the external interactions are recorded, and the replay can only literally replay what already happened.
You cannot change a variable or edit the code and branch off into a different execution path, that talks to the outside world differently; the rest of the world did not get rewound with the program.
And Factor and many implementations of Lisp too!
That's more or less the point the author makes on the next line: "Different tools support different features and have different limitations." The ideal debugger does not necessarily make for the most ideal programming environment, just as a plane made out of steel is great for structural stability but may not be a good plane.
It is great that it exists, but it isn't Smalltalk where at any moment the debugger can come up, you can change the world and hit continue without any issues, with all live instances updated to the latest set of changes.
I would say that for the most part I am a printf-style debugger. I remember reading some years ago (on HN, I believe) about time travel debugging in - I believe - C# and I was really impressed but I don’t code in C# so it soon left my mind.
I have a deep appreciation for mastering one’s toolset. I can’t think of the number of times that I learned something new about a tool (language, editor, shell, browser, whatever) that I use daily that changes my workflow - it has happened so many times. And as with all things code, sometimes new features are added.
I am going to try to make a point to re-read this article later and to visit each link and glean what I can.
I mostly code in Go. I wonder, does anybody know how much of this stuff might be supported there?
The Delve debugger for Go supports debugging rr traces: https://github.com/go-delve/delve/blob/master/Documentation/...
Undo (who I work for) maintain a fork that debugs our LiveRecorder recordings: https://docs.undo.io/GoDelve.html
Either rr (https://rr-project.org/) or our UDB debugger (https://undo.io/solutions/products/udb/) can do some time travel debugging of Go programs via GDB's built-in support for Go. I believe its weakness is in support for goroutines, since they don't map well onto its idea of how programs run.
- being able to open the debugger directly from the program. What the "debugger" command does in JavaScript. Conditional breakpoints are easier to work with if they can be directly included in the source code
- be able to open another debugger on top of the code I see in the debugger. I'm doing the stepping, I'm at a certain point in the method, and I can simply mark the part of the method code that has already been executed or is yet to be executed and start stepping it with the next debugger. This is especially useful for code without side effects. Then I can continue stepping the original method.
- be able to have multiple debuggers open and compare their status
- to have more freedom in the visualization of values and objects. Having them open in other windows independent of the original debugger, being able to interact with them using code (which can be debugged independently)
- be able to save the state of the application and debuggers so that for very hard-to-reproduce errors, I can easily recover the hard-to-retrieve state and experiment with it repeatedly without worrying that I won't get the state again right away
The following code (probably, haven't tested it) allows to create a conditional breakpoint only when a certain method is called by tracing down the call stack
debuggerIsInMethod: aMethodName
ctx := GRPlatform current thisContext. "grab callstack"
[ctx isNotNil] whileTrue: [
(aMethodName asSymbol = ctx method selector) ifTrue: [ ^ true ].
ctx := ctx sender ] ].
^ false
someMethod
"do work"
debuggerIsInYourMethod: 'aMethodHigherUpTheCallStack' ifTrue: [ 1 halt ].
"do more work"
Pernosco (which builds on rr) has some really powerful state-recording and state-sharing UI for long-term and collaborative debugging. https://pernos.co/about/notebook/
How do I learn to make such tools? From what I heard everything in the video is built in-house including the language and compiler. Although I understand many game studios do similar stuffs this is by far the most impressive I heard about.
Setting a data breakpoint, then rewinding time, then fixing the error in code (while game is running), hot-reloading code and continuing the run with it fixed. All from a recording of a panic dump from another user.
This is next level stuff. Extremely impressive.
Naturally none of them have had raw gdb, rather modern graphical debuggers.
QEMU does this too. This plus its GDB stub means one can time-travel-debug pretty much anything on any emulated architecture.
I don't use it only for debugging; it's also a great tool to help anyone understand a complex codebase like ours.
Not sure how or why the code reaches a particular function? Set a breakpoint and then look at the call stack. It's all right there.
And if you want to write some experimental code that will execute in the context of that breakpoint, just open the Python console. You can import any modules you need and test your new code immediately.
I also appreciate good logging. Especially when there was a problem yesterday in one of our customer's drive-thrus. I can't attach a debugger, but I've helped make sure we have solid information in the store logs.
Just the other day I was working on some testing code that would send an MQTT message to one of our services, and then it did a sleep(n) call to wait for that service to complete our task. The number of seconds to sleep was pure trial and error. Sleep too long and the tests take too much time to run. Don't sleep long enough and the tests become unreliable.
So I figured I would add a bit of code to that service to send another MQTT event back to the test code after completing its task. Instead of sleeping, the test code would just wait for that signal.
But where to put that MQTT call in the target service? It's a lot of rather complex code. So I slathered that service with hundreds of log.info("abcxyz") calls, every place I could find to put one.
It quickly became apparent from the log where the service completed the task and it stopped logging other messages. That's where I added the "I'm done" signal.
One thing that helped here is that our logging library adds the line number where each log.info() was called. So I didn't have to customize that log message, I could just copy and paste the same log.info() call hundreds of places.
I think every developer should learn how to effectively use both a debugger and logging. Otherwise you're working with one hand tied behind your back.
We had a conversation about this last week, including a link to an article with a title I found astonishing - "Debuggers are for Losers":
It allows you to learn them very fast.
VS has great debugger for C# which allows you to do a lot of stuff when running the program, so sometimes I even develop the soft while having debugger and breakpoints attached
https://news.ycombinator.com/item?id=29387515
Beware: you may get criticized for it as I was in that thread... :-)
Being able to look at state step by step without having to stop, add log statements, recompile and go back are too slow (for me).
What concerns me more is that is that I end up working with contractors with 5+ years who don't know how to set up a debugger for the code they are working on.
And that concerns me. It's not OR logging OR debugging. It's both. You use the best tool for the job.
An extremely common failure mode for less experienced developers is messing about in a debugger all afternoon for a problem that should be solvable in 5 minutes. It's a very slow way to learn.
Some things are really amenable to the debugger, especially simple bugs. Really especially ones that should have been caught by static analysis anyway but that's a separate issue.
The issue is that firing up a debugger can easily become a fishing expedition. If you don't understand the system behavior and you don't know where the real problem is, you can end up manually stepping through many many layers of a system trying to see what it's doing. Add asynchonicity and this can take hours before you get to the 20 lines of logic that is the actual problem.
Ideally you have better logging and introspection, so the problem is caught in a way that leads you to those 20 lines immediately.
And of course design can make a huge difference in how understandable and localizable things are.
The problem a lot of inexperienced developers run into is if they are used to having a featureful debugger available it can become the only tool they have in their toolbox, and they are rewarded by how quickly it helps them fix the kind of mistakes their inexperience leads them to making a lot of. When they run into a real issue, they can be hitting "step" for hours....
The other pernicious side of this particular coin is that it can lead to localized understanding of the problem and make it really easy to apply a localized solution. With inexperienced people this can quickly lead to band-aid solutions all over the place. With any luck someone more experienced is looking it over and pointing out they should go after the actual problems, but left unchecked it can make a real mess.
It's amazing how many developers go sometimes a decade before they learn to use a debugger.
The reason I don't usually use (/need) a debugger is that I know how the code should behave, because I thought about it in advance. Or, if it is not mine, I expect it to be readable and maintainable, otherwise I push for cleanup instead. If it is written in small manageable chunks, covered with tests and if it has good logging (which is necessary anyway - there will probably be no debugger available in production), I simply don't see the added value of a debugger. If it is not, it is not a debugger that is missing. :-)
That said, it is still a valuable learning tool because it helps understand the flow of the code, and it helps when refactoring spaghetti code. Reverse engineering also comes to mind... But other than that I can't be bothered to set it up either.
That said, I still think there's quite a bit of improvement to be made, which is why I started building a new debugger for myself which puts a lot more focus on breakpoint-less workflow, speed of iteration and scripting ability: (demo) https://www.youtube.com/watch?v=qJYoqfTfuQk It's geared mainly for gamedev, but I also do use it many times to e.g debug itself.
If you already have a good idea of the control flow (and thus, expected behavior), but there's some minute detail that goes wrong, you just need a few strategically placed prints to see the state evolution. If the problem is control flow related, you might first need to get a grasp of that before going into the weeds.
1. Use TUI mode with gbdtui
2. Install the packages that will give you syntax coloring in TUI mode (source-higlight something)
3. learn about Ctrl+L or alias `refresh` to `r` t redraw after your program prints if it does
Bonus 4. gdb uses readline just like bash and rlwrap so its command line is configurable with the ~/.inputrc file. I personally enjoy the vi mode. Tons of other options.
Bonus 5. alias gdbtui to `gdb -q -ex start --args` so it shuts up and set a temp breakpoint on you main
Then you basically have a c interpreter under your hand. I remember writing a function for my class that would dump a .dot file based on the state of my tree that I could just call anytime in gdb and see my tree update in xdot. Wonderful time.
It may not exist, but this demo has got to be the closest thing to the perfect debugger I've ever seen: https://youtu.be/72y2EC5fkcE
That's why I'm creating the next generation debugger for Rails. https://callstacking.com/
I'm not an embedded dev neither am I a debugger developer but I'm playing with toy OS dev so just curious.
and the debugger from Keil (used to be independent, now part of ARM): https://www2.keil.com/mdk5/debug
The fact that you can pause execution and start writing commands to execute on the fly in the debug console is VERY POWERFUL.
"The ups Debugger" <https://ups.sourceforge.net/>
This is the basic Sour Grapes concept.
E.g. you cannot really watch variable changes. There are some workarounds, like writing a custom __setattr__ or __setattribute__ in case of an object, or checking all STORE_* operations. https://youtrack.jetbrains.com/issue/PY-30387 https://github.com/gaogaotiantian/watchpoints
Reverse debugging is also sth I would like to have, and there are a few projects to support this, but it's not really well supported in standard CPython. https://foss.heptapod.net/pypy/revdb https://pytrace.com/
1. Cannot interrupt a running program to get debugger prompt, you have to code that functionality in yourself.
2. Cannot deal with threads or multiple child processes because it gets confused by where its output should go, and so it appears to be stuck, prompt doesn't show up, or input isn't being accepted.
3. If someone writes an overly general except, debugger won't ever be called because in order to be called it relies on exceptions.
4. Debugger cannot evaluate comprehensions correctly due to scoping issues (not as much of a debugger fault as much of the language fault).
5. Debugger cannot consistently modify variables inside the function call. Depending on circumstances if you execute an assignment statement in debugger, the value may or may not be set to the variable in current stack frame.
6. Debugger doesn't step into iterator implementation (iirc, I might be confusing with tracing).
> People think I'm a nice guy, and the fact is that I'm a scheming, conniving bastard who doesn't care for any hurt feelings or lost hours of work if it just results in what I consider to be a better system.
Looks like there is no universal golden path. Differing environments make different approaches effective.
Also, Linus is writing this 22 and a half years ago, where the capabilities of debuggers were... far, far less. Time-travelling debuggers is really a game changer, just having the ability to travel back in time to figure out who set the value that causes the code to crash. Hot reload is also a wonderful thing (unfortunately, the fragmentation of tooling in Linux makes getting this working properly very difficult).
That said, much benefit is gained by writing the complicated bits in such a way that they can be tested/debugged/examined independently on a host system.
Be happy if you can flash a led... happier if you have two speeds... and Nirvana if it is a multi-color led.
Intense jealousy of your colleagues who have TWO leds on their boards!
Core dumps have existed forever. They give you a stack trace and register values at the time of crash. Even better, you don't need a debugger running at the time of crash and you can dig into dumps sent from nontechnical users.
Sure, Bret Victor's demo was cool. But time travel debugging is so completely oversold at this point that I can't take anyone seriously that mentions it.
The most useful aspect of time-travel debugging for me, personally, has been when the test case that causes a crash is refusing to be reduced, and the function that crashes does so on like the 453rd time it's called. Jumping straight to the crash, then reverse-continuing to a break point cuts out so much time of debugging (especially because it saves you if you accidentally continue the breakpoint one too many times; otherwise, you'd have to start the entire, tedious process from the beginning).
Now, imagine you did do that manually and spent a lot of time finding that location. In many cases, that only gets you one step closer to the root cause, and you have to repeat the operation multiple times. Yes, you _can_ do that without record-and-replay, but do you really want to? Do you want to spend hours doing something that could take you minutes?
(And that's not even mentioning even worse cases, where the value you're tracking goes between processes via IPC)
With a time travel recording I can put a watchpoint (aka data breakpoint) on the state that is supposedly impossible, and reverse-continue back to see exactly where it got set. (And repeat as required.) It really is very powerful.
Admittedly there are situations in which it's not practical to get a recording, but when you can... almost any bug becomes trivial.
learning how to undo my last step would save so much time. right now, i have to anticipate a risky step and run that line in the debugger console.
I don't think there is a solution for PyCharm.
https://learn.microsoft.com/en-us/azure/azure-monitor/app/ap...
Or Java Flight Recorder,
https://developers.redhat.com/blog/2021/01/25/introduction-t...
This is a good list, but one thing missing: catchpoints. Particularly useful with time travel debugging. catch throw + reverse-continue immediately tells you who threw an exception, and you can continue debugging from there.
What I do is I encode conditional breakpoints in the source code, (compile and) run the program with a debugger attached. The nice thing is you can have complex conditions using all kinds of functions from your surrounding code and you can have them permanently, even check them into the VCS. It is kind of like placing asserts which don't panic but trap into the debugger and they can be globally enabled / disabled.
We used GDB's conditional breakpoint bytecode https://sourceware.org/gdb/onlinedocs/gdb/General-Bytecode-D... to get a speedup in the thousands of times vs plain conditional breakpoints.
That works for us because we've got in-process agent code that can evaluate the breakpoint condition without trapping. It should be possible to do this in other debuggers with a bit of work, though we have the advantage of in-process virtualisation to help hide this computation from the process.
And why can't we create interfaces to debuggers so that other text editors can integrate with them, much like how we do we LSPs?
*EDIT*
Thanks for all the responses! I've now heard from multiple sources that debugging on Linux is unpleasant, and it seems like the whole process is challenging regardless of the platform.
Painfully. If you're on Linux, you get to use a mixture of poorly-documented (e.g., ptrace) and undocumented (e.g., r_debug) features to figure out the state of the program. Combine this with the debugging symbols provided by the compiler (DWARF), which is actually a complicated state machine to try to encode sufficient details of the source language, and careful reading of the specification makes you throw it out the window and just rely on doing a well enough job to keep compatibility with gdb.
> Why can't more compiled languages integrate debuggers inside of them so that you can debug the program without using a separate tool?
Because it's really painful to make debugging work properly. At least in the Unix world, the norm is for all of these tools to be developed as separate projects, and the interfaces that the operating system and the standard library and the linker and the compiler and the debugger and the IDE all use to talk to each other are not well-defined enough to make things work well.
> And why can't we create interfaces to debuggers so that other text editors can integrate with them, much like how we do we LSPs?
LSPs have the advantage of needing to communicate relatively little information. At its core, you need to communicate syntax highlighting, autocomplete, typing, and cross-referencing. You can build some fancier stuff on top of that information, but it's easily stuffed in a single box.
Debuggers need to do more things. Fundamentally, they need to be able to precisely correlate the state of a generated build artifact to the original source code of the program. This includes obviously things like knowing what line a given code address refers to (this is an M-N mapping), or what value lives in a given register or stack location (again an M-N mapping). But you also need to be able to understand the ABI of the source level data. This means you can't just box it as "describe language details to me", you also have to have the tools that know how to map language details to binary details. And that's a combinatorial explosion problem.
It's true that coming up with an interface for an abstract debugger is harder, but it's not impossible. Microsoft created Debug Adapter Protocol (https://microsoft.github.io/debug-adapter-protocol/), which is conceptually similar to LSP. It's not perfect, but covers most basic operations pretty well, while leaving to the debugger to deal with the implementation details.
The protocol you've linked (or the similar gdbserver protocol) essentially implements an IDE <-> debugger mapping. Well, most of one: everything is basically being passed as strings, so if you want smart stuff, you kind of have to build in the smart stuff yourself. It doesn't help the other parts of the process; if you want to build a new gdb, you have to do all the parsing of debug info for C, C++, Drustzig, etc. yourself... and you have to integrate the compiler expression stuff yourself. If you want to build a better time-traveling debugger, or a better crash debug format, or something similar, well, the gdb remote protocol lets gdb talk to your tool so all you have to implement is essentially low-level program state (e.g., enumerate registers of a thread, read/write arbitrary memory locations, etc.). But this isn't covered by the thing you've listed either, and it still relies on the debugger supporting a particular protocol.
> I can implement the LSP and get support for IDEs essentially for free I mean, technically same is true for DAP... You can implement DAP and get support for IDE for free. But I agree that in general case implementing a good debugger is harder than implementing a good language server.
If Drustzig requires a special debugger (e.g. because it uses acustom format of debug information), then you'd need to implement it, yes. However, existing debuggers can support new languages relatively easy if those follow standard conventions (e.g. use PDB and DWARF). For example, Rust support in LLDB basically comes down to a custom demangler.
Again, I'm not saying that DAP is perfect and solves debugging, but IMO it's a step in the right direction. Make it popular, make it extensible. Debuggers can be mostly language agnostic (within reasonable bounds), but they don't _have_ to be.
It has some rough edges sure but I'm using it successfully with C and Rust. If you have tried but still don't like it what's your use-case?
A debugger is basically a very complicated exception handler that can, with the help of the kernel, intercept exceptions from other processes and access their memory. (This is for Windows, but I would guess that Linux is about the same.)
When process X wants to debug process Y:
1. Process X calls into the kernel and says it wants to debug process Y.
2. The kernel verifies that process X is allowed to do that. (You wouldn't want a low privileged user to be able to debug a service, right?)
3. The kernel triggers a debug break exception in process Y, usually.
4. Process X goes into a loop where it asks the kernel for the next exception from process Y, which is a blocking call until Y has an exception.
5. The kernel's exception handler catches the exception and passes details about it back to process X.
6. While process X is in control, process Y is suspended and process X can use other kernel calls to read and write process Y's memory.
7. When process X is done doing whatever, it tells the kernel to continue the previous event and asks for the next exception.
Process X will have, of course, loaded some libraries that help it navigate the structures in memory in process Y, starting with something at a well-known address like the PEB. That lets it do things like enumerate threads and loaded modules, find symbols, etc.
Relevant win32 calls are:
WaitForDebugEvent, ContinueDebugEvent, ReadProcessMemory, WriteProcessMemory
(Standard caveat that I haven't actually written one of these things in 20 years applies. Maybe there's new stuff now.)
we have pretty magnificent tools for the mapping to a shared semantics of syntax-
tree-sitter :: any syntax -> 1 semantics
But we don't have great tools for mapping (1 semantics -> any syntax) - in some sense, the inverse of tree-sitter:
inverse-tree-sitter :: 1 semantics -> any syntax
The key impact of this is that anybody from some individual language could add a plugin, which could apply to every supported language that uses the subset of features that apply to that plugin.
Think about the network effects of a language which has a large universe of plugins - now think about the even greater network effects if it could be a union of supported languages - that's strictly at least as good, if not vastly more network-effectful. So I think the benefit to people would increase rapidly if people contributed to it across many different languages, so it could be a good collaborative tool.
I bring this up under this discussion because the debugger would be the exact best place to apply something like the inverse of tree-sitter
pretend I'm not ignoring this would likely lead to a strangely organized library with lots of potential spots of inter-language interface annoyances, so it would not probably be strictly better... :)
Who? Who does say that? Freshman students who barely started coding?
>I suspect many of them think that debuggers can only put breakpoints on certain lines, step-step-step through the code, and check variable values.
That alone provides enough value to know debuggers are useful.
Example from this thread: https://news.ycombinator.com/item?id=35098434
That aspect of debuggers hasn't changed in 23 years, because that's the whole point of debuggers anyway.
Some projects are better off with a debugger and some developers are more productive with them. But also the other end of the spectrum exists: projects that are better off without and developers who are more productive without them. Not liking debuggers don't mean someone is stupid or inexperienced.
I'd even dare say that's more likely an experienced developer to not use debuggers as much as people earlier in their careers. Seasoned developers working for a long time in the same code base are more likely to have insight of what's going on without stepping through the process execution.
What a bizarre take. Just today I had to use a debugger to figure out where there was a deadlock in our program and why. No amount of printfs would have made that speedy. In the debugger I just stop the program once it's hung and look at all the threads to see which threads are holding which locks and which are waiting to obtain locks. From there I can now see it's a lock inversion because we weren't being careful about the order of taking our locks in a few places. That lead to doing a wider check to see if there were other similar cases, and whether we needed to rethink anything in particular.
The debugger generally gives me an overview of the whole program in a way that a bunch of printf statements can't. (Which isn't to say that printf doesn't have a place in debugging - it certainly does.)
It depends on individual’s strengths and weakness as well. Neither way is going to be better for everyone. Each person has to find out what works for them.
Said differently by other old beards in this classic of Ken Thompson & Rob Pike: https://www.informit.com/articles/article.aspx?p=1941206
> When something went wrong, I'd reflexively start to dig in to the problem, examining stack traces, sticking in print statements, invoking a debugger, and so on. But Ken would just stand and think, ignoring me and the code we'd just written. After a while I noticed a pattern: Ken would often understand the problem before I would, and would suddenly announce, "I know what's wrong." He was usually correct. I realized that Ken was building a mental model of the code and when something broke it was an error in the model. By thinking about how that problem could happen, he'd intuit where the model was wrong or where our code must not be satisfying the model.
> A year or two after I'd joined the Labs, I was pair programming with Ken Thompson [...] Ken taught me that thinking before debugging is extremely important. If you dive into the bug, you tend to fix the local issue in the code, but if you think about the bug first, how the bug came to be, you often find and correct a higher-level problem in the code that will improve the design and prevent further bugs.¶ I recognize this is largely a matter of style. Some people insist on line-by-line tool-driven debugging for everything. But I now believe that thinking—without looking at the code—is the best debugging tool of all, because it leads to better software.
But I still default to use printf/log/tracing debugging style because it's often easier to get some signal under the conditions I care about and are hard to reproduce where it's hard to attach a debugger.
Or you write debugger-legible code, you use the debugger, and you get feedback when the code you wrote confuses the debugger, so you refactor it to be easier to diagnose the problem, then you commit those changes.
Supports a bunch of stuff described there and more.
I will never forget the first time I changed a few lines of assembly of a running game (starsiege tribes, around 2005), and then I saw friend-or-foe indicators through walls.
That kind of "hot reloading" was special.
Certainly tools like App Dynamics and Datadog enable this app stack tracing, and Docker Compose enables composing services.
To be able to play back transactions backwards and forwards among services would be a dream.
Wondering if any folks have played with this kind of debugging?
Time travel is great for understanding someone else's large program, so that you can see where it goes as it is running. In large projects the instruction pointer is hop around madly throughout the code, and seeing where it goes like the path of a bird is rather handy.
You can use rr or LiveRecorder (commercial product, which I work on) to generate the recording non-interactively then debug it "locally". Avoids the need to set up a client/server configuration, so long as you don't need to modify variable values at runtime, etc.
Nowadays only if I am on bare bones installation, I reach out for emacs or vim, on that order.
The downside of debugging is obvious: it's not enabled in your production programs (although it could be).
Logging OTOH is always on, and it has to be good enough to let you figure out what happened on a remote machine, in a program whose state you can't recreate anymore.
This obviously makes it good enough for debugging programs running locally, and makes reaching for a debugger a rare occurrence. GUIs for debuggers of course lag behind, so you need to refresh your memory of the step, step over and breakpoint instructions, etc. on every one of these rare occasions.
And then they don't always work. Printing more complex data structures is often an unholy mess of pointers to strings and private class fields. Evaluating expressions will end up failing or crashing your program. The symbol you're trying to break on doesn't exist? But why? This makes debuggers unusable for a mediocre developer.
> a good debugger supports different kinds of breakpoints, offers rich data visualization capabilities, has a REPL for executing expressions, can show the dependencies between threads and control their execution, pick up changes in the source code and apply them without restarting the program, can step through the code backward and rewind the program state to any point in history, and even record the entire program execution and visualize control flow and data flow history.
TL;DR: That's fantastic! But if you need any of it you're already "doing it wrong".
Context: I've recently been using a compiler for a C-like language that targets a simple 64-bit VM, the point being there's no debugger for the stack (the VM is written in Rust; I could put a debugger on that and just deal with the extra level of abstraction using e.g. features like those described above.)
So how do you cope?
Design simple and robust systems that can be understood in action easily via printf/log. Use tried-and-true off-the-shelf components (including algorithms and datastructures.) Write in small increments, compile often, never proceed without complete confidence in your understanding of the system. When the inevitable bugs appear, and they can't be discovered through just thought or a rereading of the code, then you bisect: often literally (LoC) but also conceptually. Solve the puzzle and eliminate the places where puzzles can hide in the first place.
I quite literally see how a having a debugger would lead to worse code. (Even if you don't use it.) This isn't news btw:
> Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?
-- Brian Kernighan, 1974
(The feeling is different: like instead of a wild adventure, programming this way feels like gardening in an orderly and well-kept garden.)
> This is normal...
That's the part that I have a problem with. We have leavened our entire complex civilization with software, it's important to get it right.
I do 3D meshes programming. The amount of vertices, planes and other geometrical entities that I need to operate with in my algorithms is too big for me to do printf debugging. I can't just look on a long list of 3D vertex coordinates and visualize the mesh they make in my head. Moreover, I had to come with my own 3D visualization tool, once I got fed up with pen and paper for converting 3D coordinates to actual meshes, or using Blender for that. For me, a debugger (and debugging tools in general) is irreplaceable. I'm not saying my field is the most complex one, because it clearly isn't - but in that field, I can't think of any other way I could do what I do.
That's my biggest criticism of debuggers having used both approaches: you forget that sometimes you need new tools. Whereas with prints you're constantly building new instrumentation for yourself.
What does this mean? Formatting text differently is still text, which the parent post was just explaining doesn't cut it.
Undo is another one for Linux programs, and I think I've seen developers from them post here.
The most sophisticated debuggers seem to target C or Lisp. But lately I don't use either.
I've never gotten time-travel debugging in gdb to work. And it's been out for more than 10 years at this point. The last time was 2 years ago, so I forget the details. I do love tracepoints in gdb.
I've been watching https://rr-project.org for a while. The instructions still say, "build from source".
I looked at LLDB after your previous comment. Got it installed, but I can't actually get it to set a breakpoint on Linux. The docs seem optimized for Mac.
I have no doubt that once you get something set up just right you can do great things with it. But the power to weight ratio seems totally out of whack. Part of the goal of my projects recently has been to show that you can get a bunch of features far more simply if you build on a base of logging. If the programmer is willing to modify the program, a single tool can help debug programs in a wide variety of languages. They just have to follow a common and fairly simple protocol.
The big drawback of my approaches is that they don't scale for long runs of huge codebases. That hasn't been an issue for most programs I ever want to debug. Why should I pay the complexity costs of debugging gcc or Firefox for small programs.
rr should be in most distros at this point. (It's been in Debian since at least 2014).
[1] Though I had to use sudo a few times, reboot, see some scary warning that it might not work reliably, read https://github.com/rr-debugger/rr/wiki/Will-rr-work-on-my-sy... and https://github.com/rr-debugger/rr/wiki/Zen.
Indeed, using GDB's version of tracepoints (dprintf - https://doc.ecoscentric.com/gnutools/doc/gdb/Dynamic-Printf....) is really powerful and replaying a trace with these installed is to generate "logs I wish I'd had" is exciting.
It also has potential use if:
* you'd like to diff two executions (e.g. one successful, one failing) - you probably can't just compare raw execution as there'll be uninteresting variation but comparing extended logs could be useful
* you are debugging something at a customer site - they might not let you debug directly but you could iterate on plain text logs by shipping them additional tracepoints to run on a recording
We thought this was useful enough that we implemented a tool with its own DSL for simply specifying post factor logging "probes": https://docs.undo.io/PostFailureLogging.html
I'm arguing, "don't just use debuggers."
I elaborate more on this argument in the first 2 minutes of this 4-minute video: https://handmade.network/snippet/1561
As for "forgetting", I don't think I forgot, because, well, I did make a new tool.
The main thing that I wanted to address in the parent comment is that needing tools to debug your code somehow means that the code is a mess - no, sometimes it doesn't.
Ah. I totally agree with that. Or at least, it's the sort of mess I don't know how to avoid yet.
I do use debugger. During big fixing, I need answers to various questions, such as: on what halfspace does a vertex lie? Is this point inside that polygon? What is the distance between two points? - and so on. If I didn't have a debugger, I'd have to stop the program, find the distinguishing features of the entities I'm interested in again - and this is often the hardest part, since the same code can be called thousands of times with different data - put printfs, rebuild, relaunch and prepare for another cycle. With a debugger, I just put these questions into LLDB queries - and I get answers. It is so much faster.
Would it be possible for me to do my job without all that, using debug prints only? Theoretically, yes. Would it be practical? Absolutely not.
The fellow I learned that from was apenwarr, FWIW.