I do not use a debugger (2016)
lemire.me
lemire.me
It _may_ tell us something about debuggers. Probably that they are not as useful when it comes to OS kernels.
Just as well, it _may_ tell us something about Linus Torvalds. Probably what tools he is more proficient with, or likes more.
> Debuggers were conceived in an era where we worked on moderately small projects, with simple processors (no thread, no out-of-order execution), simple compilers, relatively small problems and no formal testing.
Unix was conceived in another era too, but modern Unices are totally different and have been adapted to modern times. So have debuggers. When I debug a Java program in IntelliJ I can stop and run threads as I like; OOO execution is transparent; so is both high-level compilation (javac) and low-level JIT compilation by the JVM.
> in order to catch bugs, we often need to be able to run with sufficiently large and representative data sets. When we’re at this point, the debugger is usually a crude tool to use
You'd typically use a debugger to understand why one of the tests with your large dataset fails, after you have already noticed that there is a bug. You don't use a debugger to notice that a bug exists at all.
This article really has a point, but it's buried under crude logic, and the elephant in the room is that "debuggers vs. tests" is a false dichotomy.
Remote debugging a kernel on another machine is certainly useful and always a bit magical for me.
It's sort of funny that the author even admits "Ok, well, I do use a debugger but only when it makes sense". Like, are there devs using debuggers when they don't make sense? I don't often drop into a debugger just for fun.
Debuggers (and, particularly the java debugger) are amazing tools for quickly figuring out why things went wrong. I rarely step through code (though sometimes that's useful) but I frequently will drop a breakpoint on an exception and then jump around the call stack to see exactly how a particular exception developed. With a debugger, I get all the inputs for everything up and down the stack. I can pop the stack and replay the exception over and over again. I can evaluate code in the current scope to test and see what's going wrong.
Particularly with really complex software stacks, debuggers are essential.
Consider, for example, this error case I ran into.
Some code took a view on a map. Unfortunately for us, modifications to that view are both allowed AND mutated the underlying map. When the code that produced that view called other code, at about 30 layers down some code said "If there's not a value here, add the value 0". This caused the unusual state where with apparently the same input data when you first called a method, it would pass no problem. The second time, it would error out.
How on earth would you discover this behavior without a debugger? The root method where this all happened was already 20 layers deep in this fairly complicated piece of software (yay finance!). A simple unit test would show everything working as expected, it's not until that second run of the same code that things blew up.
Without actually catching the fact that the inputs were mutated from an earlier invocation there's really no way to know what went wrong. A debugger made that obvious.
[1] http://www.linusakesson.net/programming/syntaxhighlighting/
Yes. Older developers tend to debug with console/echo/print etc. Somewhere along the line this was deemed “bad practice” and now there are devs who will exclusively use the debugger for everything because they were taught it’s the “correct” way.
People will do what they’re taught. I grew up with console/echo/print so that’s how I roll. I never use the debugger, to my detriment. Conversely, people who grew up exclusively using the debugger will use the debugger for everything even when it would be way faster to just dump some data.
Even primitive debuggers like gdb are not intended to be used like that. You theorize a failure mode, identify the proximal causes, then set breakpoints at the earliest suspicious points. You then examine the actual state to determine if your mental model of the code matches. You can then single-step to verify that the suspicious piece of code does or does not do the correct thing.
If you are feeling really adventurous and are willing to try technology that is only 20 years old you can use time travel debugging. Then you do not even need to think about how you debug. You just hit the bug then run backwards to the cause of the error. Then you examine the entire state of the universe at the time trivializing basically every bug.
You can use even newer stuff like high grade visualization of the entire program execution which trivializes bug detection, performance enhancement, and system understanding.
These articles always seem stuck 50 years in the past and even then seem to always be arguing that hammers suck because they keep trying to hit the nail with the handle.
Very interesting... what is the gdb-reverse-debugging equivalent for high grade visualizations of, for ex. cpp or python software? Something that's free, open source, generalizable
By GHS, developers of TimeMachine (TimeMachine created in ~2003), for C/C++: https://www.ghs.com/products/MULTI_IDE.html see “History viewer”
By the developers of rr (rr created in ~2013) for C/C++: https://pernos.co/
By former GHS employees for Python: https://functiontrace.com/
Usable FOSS time travel debugging is new enough that no follow on FOSS visualization has been developed as far as I know.
It’s not that I enjoy stepping through code line by line, I rely on a debugger to verify if my assumptions about “starting conditions” of a certain piece of code are correct and, if so, at which point my understanding of how things should be no longer reflect reality.
This can all be done with printf, if you knew ahead of time what you want to print. But most of the time I don’t, the whole point is that I do interactive exploration of the program state (“evaluate expression” is the main feature of a good debugger). When I find a problem I dig deeper, which is a lot faster within a debugger session than any recompilation loop I’ve ever seen (even in Go).
In similar spirit I play with SQL queries when given a data set that I need to extract something from.
I do this even with high-level languages like Python.
Absolutely. It's not that I use a debugger that often, but when it's needed and a good one isn't available, I'll be hitting my head against the wall.
Languages without excellent debuggers (and profilers) have no place in production code. Because when you need the tools, you need them.
If I sprinkle a couple of prints and my hypothesis about what state I need to see is wrong, then I have to do another build and run cycle to get more information. If I'm stopped at a breakpoint I can dynamically and interactively root around without having to go through the whole scenario again.
Maybe this is just a different mode of using the debugger than what's being argued against. I will say that stepping through is really only fruitful in my experience when there's an extremely narrow region of code that I've identified as causing the problem.
In any event, standard console logging is an abomination and there are much better encoded logging techniques that are exactly as usable and just as easy to add to your code, while not introducing material timing skew.
Print debugging only captures what you think to capture. Time travel debugging captures everything. And with a proper implementation the overhead is minor.
I use debuggers sparingly, I think they can help in some cases but most of the time I feel much better reading code and traces. Especially when debugging difficult async code.
Being condescending with coders who have different habits than yours is the issue.
There is no "one true way" of doing things, even in our beloved craft.
I generally reach for a debugger when code is complex enough that I don't know where to begin putting my prints. But often end up reducing to prints and other output when I get close to the issue.
That said. For buggy code, where the bug appears seldomly, conditional breakpoints are a lifesaver.
I'm not big on using a debugger but when all else fails, having access to one can be the difference between a swiftly identified and resolved bug and weeks or months of hair pulling and stress.
It tells us that its possible to program without a debugger It tells us absolutely nothing about the utility of doing so because anecdotal preference cannot possibly reveal this.
> Will your debugger scale to hundreds of processors and terabytes of data, with trillions of closely related instructions? I’d rather not take the risk.
Almost nothing almost anyone is working on needs to work on TB of data across hundreds of processors to reveal a logic error or in fact at all.
The article could profitably be shortened to "I feel like debuggers are a bad investment of my time because I would prefer to spend that time revising and writing tests because my mental model of the tech I work with is sufficiently good that I rarely need to poke the code manually and I have confused my own preference and circumstances with universal truth because that's what humans do"
That being said, once I fully understand the entire stack and how we're using it. I can abuse the hell out of it and make it do anything I want it to do; even beyond anything the original authors ever considered.
Most people only take the time to get surface-level knowledge of the entire stack, and either re-implement something they can get for free, or don't take full advantage of what the stack gives them.
That being said, getting that knowledge without a debugger is nearly impossible.
Somewhat to my surprise, I was able to use gdb to find out why a multi-thread program Rust had a 100x performance degradation under Wine. The trouble was down in Wine's DLLs which emulate Windows. The memory allocator is vulnerable to futex congestion. They layered spinlocks 3 deep, and when enough threads are allocating, all the CPUs spend most of their time in spinlocks. The Wine devs acknowledge the problem but are still struggling with a fix. I would have used a profiler, but "tracy" under Wine is broken at the moment.
I used a debugger last year when a C++ JPEG 2000 decoder called from Rust was crashing. Valgrind was more helpful, though. Once someone got the decoder to fail under valgrind, they were able to fix the problem.
The year before that, I used gdb to find a bug in the Rend3 graphics library. That's in Rust, but has unsafe code. The bug was in the unsafe code, of course.
There are times like this when the high-level language abstraction has broken down, and you need to delve into the machinery underneath. But that doesn't come up too often in modern languages.
The one thing I agree about is that debuggers got stuck in a point in time and they cannot do much more than set a breakpoint and trace some code (with exceptions) - but this is not a limitation of the concept, it's rather that the tools we have today often fail to help us. We resort to using printf() instead because then we're looking at just one variable and when it's printed, rather than stopping the whole program or hitting the continue shortcut 50 times.
It's why I've been researching the topic quite a bit myself with some other ideas than "stop the whole thing and look at state". One thing I found is that if the debugger is not integrated into your workflow (it's not working by default after you compile/run), you'll be much less likely to actually use it. Another way to look at it is: would you still not utilize the information the debugger displays, even if it displayed the state at a line you highlighted immediately?
Debuggers are extremely powerful in the correct situation. If you can reproduce an issue in a controlled fashion, debuggers are nuts. In many cases, you drop a breakpoint where the exception/panic/segfault occurs, trigger your reproduction, and the current context + variable view just throws the problem in your face. If that fails, you breakpoint earlier and that's usually it. I've very much changed my opinion there: If you have a reproducible problem, and the source code, use a debugger. It's better.
On the other hand, good logging and tracing are better in other cases. If we call a debugger spear fishing - targeting a concrete target specifically - good logging and tracing are more of a drag net. Like, in my last job, we had a bug that effectively nulled user accounts, which possibly voided thousands of euros of investment. Except, it occured about once or twice a week, across billions and trillions of interactions with the service. This turned out to be a very, very specific synchronization issue in a rarely used and rarely contested component, but very active players were more likely to be affected by it. Very messy.
We couldn't have caught that error without just dumping all errors and a lot more events into a central bucket and then analyzing from there.
But, in a similar way, this kind of event collection in turn allows analysis like: How critical is this thing? An error occuring 20 times a day on a production system for the last 30 days probably isn't that important, else we'd have had a ticket 20 days ago. Not something a debugger can tell.
Exactly. What are the available debugging tools, and what are their pros and cons? That's a much more fruitful question.
Today I had to investigate a bug in a C program. With a conditional breakpoint in gdb I could spot the problem and devise a solution in a matter of minutes.
When I'm debugging SQL (which happens frequently), that depends. If the query is light, I often use a "Prolog-like" approach: make the expression more general until I find the clauses responsible for the bug. On the other hand, if the query is too heavy, I consider spending some time reading the expression very carefully to see if I can find the issue without incurring in a costly execution.
It all boils down to what sorts of problems you have, and which of the available solutions will be more appropriate.
It's a lot easier for single binary projects (like games) than multi- multi- target projects (web frontend, backend, etc) - plus debugging tools are less mature in those worlds.
If debuggers mostly Just Worked in that kind of environment, people would use them far more often.
And the culture is to not invest in that config
I do not use a debugger (2016) - https://news.ycombinator.com/item?id=19829435 - May 2019 (182 comments)
I do not use a debugger - https://news.ycombinator.com/item?id=11946395 - June 2016 (25 comments)
I still wonder whether it would make sense to learn more about using them, though. My reasons for not using a debugger often are primarily:
* I work with different kinds of systems. Printf or some equivalent is always available, Debuggers always require a system- and programming-language dependent setup and incur an additional learning curve.
* The hardest bugs I have encountered so far were timing-dependent and could not be reproduced when running under a debugger.
Instead I prefer to work with the printfs and some tools which work non-interactively like strace and valgrind. To me, it makes a lot of sense to look at the code when there is a problem and consider the problem in my mind, although I would not go as far as saying that this is in any way fundamentally superior to stepping through the code interactively?
I've found that higher level languages (Java/javascript/python/etc) tend to have a much better integration and experience with debuggers than C/C++ did.
Before that I also encountered similar issues at the other end of the performance spectrum: When working with parallelized C++ code that required more than 64 GiB of RAM the overhead of a debugger can make the difference between being able to run the program or not.
I still believe that in almost all cases it is possible to use a debugger even there, but it is always a significant setup required.
And in the modern languages like Java and Python there are stack traces which already greatly help with the “debugging” such that even there I rarely missed the option to step through a program. I never worked with any really complicated Java or Python codebases, though.
I think debugger vs print is more an ergonomic problem. Yes I can write print and if/else instead of using debugger and conditional breakpoint... but it's often much slower, and I need to be careful to not commit them into production code.
> Doing “something else” means (1) rethinking your code so that it is easier to maintain or less buggy (2) adding smarter tests so that, in the future, bugs are readily identified effortlessly.
I fail to see how debugger prevents me from doing this. To me it's mostly a different way to print/if. It would be very weird if using print prevents me from writing maintainable code.
"I learned to program with BASIC back when I was twelve. I would write elaborate programs and run them. Invariably, they would surprise me by failing to do what I expect. I would struggle for a time, but I’d eventually give up and just accept that whatever “bugs” I had created were there to stay."
People that dont use debuggers for serious software development had their brains develop programming ability without one. They are stuck in their ways, and generally arent aware of it. You cant teach an old dog new tricks
Personally it's all about fighting for observability. If you aren't using all the tools at your disposal you're not as effective as you could be.
> I think that my blog post is clear on what I mean by “debugger” which is interactive line-by-line execution. I specifically say that I use tools that are called debuggers. But I do not use them as debuggers.
Debuggers are clearly useful at times and don't preclude printf sprinkling, there are many tools in our belts.
Watchpoints alone justify my gdb installation.
Otherwise, I'm a great believer in logging progress through a program instead.
For my server-side stuff, I usually program in PHP, and I've always used echo(), var_dump() and die(). I've just never found all the overhead, preparing the xdebug stuff, and using a symbolic debugger, to be worth it. I find and fix bugs fairly quickly. string dumps have always worked for me.
(However, sometimes a debugger is more useful when I am trying to debug a program that I did not write and do not have the source code of.)
I certainly see now why the first instinct to step through gnarly code might not be the best one!
I started writing JS/PHP over 20 years ago. Back then I debugged with console/echo/var_dump etc. This stuck with me and is still how I debug today. Every time I see someone use the JS debugger I’m impressed by its utility. Still, I always end up reverting to console logs. It works for me and I can’t find a reason to break the habit.
real men don't use debuggers
real men don't use ide's
can program on a piece of paper with both hands tied behind their back
etc...
The actual article is much more polite than this, but I have to deal with these people, who think of themselves very highly, at work every day. Crap i'd get at the code reviews from some bizarre cult-ish schools of thought like "remove all your comments because comments in code never age well", or "logging is distracting and unnecessary", "unit tests spoil programmers so they become sloppy", people actually enjoy imposing these bizarre ideas on others.
The more relevant point, actually, is that the OP seems to do a bunch of multithreaded code and for their use case debuggers may not be utilized to their full extent. But I'm sure some half-wits at work would just read the title and run with it.
Also, I find debugging JS to be really difficult these days because the ability to set breakpoints and inspect state just doesn't work half the time in Chrome.
What I really want is a "debugger" statement I can turn on/off, at least then it'll break where I want it to.
* The logs don't have enough info (if there's even logs); and
* I can't write a quick test to reproduce the issue, either because there's no tests or because they're so overengineered that it's not worth the effort.
Given how great Java/C#/JS debuggers are, one could hope that every new language will ship with a working debugger, that it would be table stakes.
But there’s a whole generation of working programmers that started their careers in Go and were told debuggers are lame all along, so it’s hard to blame them for not using one.
As for fashion and being told what is cool - I do not give a flying fuck about propaganda. I use what I think fits me and provides best ROI (I am my own business and pay all the expenses).
Why are print statements better? Could a debugger, designed considering this, beat print statements?
Let me visualize the changes of variables over time and what code caused those changes.
Or one click in the gutter to add "log points" vs having to type "print(...)" every time.
Or even track everything and let me activate logs after the fact, as I learn where I need more granularity.
The conclusion of "print is better than debuggers" doesn't sit well with me.
The standard debugger design sucks--so we to resort to print, which sucks a bit less.
If anyone has links to fresh takes on debuggers, please share!
strace is all about logging the state over time. Of course, it's limited to system calls, and not C library stuff or algorithmic computations, but if your application is system-call dependent, such as networking or file I/O, strace can be a great tool.
Then pernosco ( omniscient debugger by Rob O Callahan and Kyle Huey ) happened - it’s incredible.