Print Debugging Should Go Away
robert.ocallahan.org
robert.ocallahan.org
I get the point this person is making, but at times (and depending on language) it's the fastest and most efficient option. "Oh, the value is not what I expected" -> fix -> "value is good after testing, can remove the debug print". It's not elegant, yes, but it does the job with zero preparation and works for every language. Now if things are so hopeless the debug print doesn't even work, then yes, get a proper debugger.
I argue that printf debugging is more difficult if you dont know _what_ is incorrect, or if you dont know where a value is being changed. Stepping through your code is a great way to find out.
To me, actual stepped debugging is like having a print statement for every single line in the code base.
Or even impossible? I use print debugging often, not even only in cases where there's no step debugging possible and/or I'm too lazy to fire up a config/software which allows debugging, but from time to time I encounter bugs which I really can't even begin to imagine how terribly awkward it would be without step debugging. For example: finding a bug in a mark&sweep garbage collector written in C which only occurred in certain LTO builds. To get a picture of the problem I stepped through assembly code while looking at certain memory locations, regsiters, the C stack, and some related variables. How would one tackle that with print debugging? Plus the print statements would affect the very thing being debugged? That should be fun. Not to mention the effort it took to even be able to reach the exact code location where all of this happened. Would almost require to randomly sprinkle print statements all over the place.
EDIT: I don't want to go so far as to say that your example can never happen, but I can't believe there is any domain where it is the norm
I don't get your point, because that's not what I suggested someone should do. I think (on rereading my original comment) my point was the exact opposite: Develop an understanding of the code through the effective use of a debugger:
> In fact, the debugger can be a fantastic way to learn the code since the dynamic behavior of a system can often be hard to infer from its static representation (looking at you WPF and XAML, or any project where architecture astronauts went hog wild with certain design patterns).
You seem to have inferred something from my comment that I did not say.
Add some instructions to print "**1**", "**2**", "**3**", ... to see how far the program goes, which branches it takes, print a few variables that look interesting. If that fails print the uninteresting variables, remove "**1**" and add some "**2a**", "**2b**", ... in between 2 and 3 ...
After a while, you discover the correct spot and the correct variable, and the error is trivial.
Looking at the effort involved: here you have to spend the time to add a bunch of print statements throughout bits of code; then compile/run the program (repeating until you've found the right place).
With a debugger: you'd have to take the time to set up the environment and learn how to use it (e.g. set breakpoints in parts where things would be interesting). But then the "look at interesting values" wherever isn't significant additional effort.
(I use both)
This is really the problem. Nobody has spent time and money to make debugging _good_ in the languages you use. Few people pay for good tools any more, and only a tiny fraction of users ever contribute improvements to the open–source tools they use (paid or otherwise). So the average debugger hasn’t gotten better since the 90s.
I do programming in multiple languages too. In those where I can use RR and Pernosco, I use them every time. In those where I can’t, I miss RR and Pernosco every single time.
* Debugging an 8 bit microcontroller in-circuit with an IDE, conditional breakpoints, step in/out/over, full stack visibility, and the ability to manually poke at values/memory/io
* Debugging an AWS lambda function with printf and high latency logs
It doesn't matter how hip, how modern, how fast, or how big the environment is. What matters is whether or not the ecosystem cares enough to make debugging nice.
Live by the sword, die by the sword.
As pg said, it's like scrubbing dishes with your fingertips: rarely discussed, highly effective.
Print debugging is like scrubbing dishes by hand, and neglecting your dishwasher.
Pernsoco is like those commercial dishwashers that blast the dishes with super–heated steam to wash them in a few minutes instead of 45.
I suppose I feel the same way with print debugging. Mostly that's all I need and the problems I have which would benefit from better tools are rare enough and diverse enough it isn't worth investing time in them.
Anyway I don't really care about some insignificant "environment" difference - the affect on daily convenience far outweighs it.
A tub setup would be more environmentally friendly considering the resources put into a dishwasher these days. Certainly drying via a towel is better then an electric heater.
Oh hell no they aren't. They're terrible at handling anything that's dried on plates.
I've never found a dishwasher that could clean truly dirty plates. NEVER. Even if you run them ten times in a row.
Of course I’m as lazy as the next person; there are dishes in my sink right now that could be put in the dishwasher, if only someone would empty the dishwasher of the dishes I washed the other day…
Even so, I can only recall one time where it didn’t get a baking dish fully clean. I just left it in the dishwasher and ran another load the next day and it was fine. Then I baked some more enchiladas in it.
You need to pre-wash everything before loading it anyway, and in the end I still need to scrub and re-wash some of the dishes in the sink.
I'd rather just do it myself from the beginning. Takes less time and effort, and comes out cleaner.
Are you using a rinse-aid? If not, you're doing it wrong.
I never pre-wash. I'll manually remove solids when I'm done eating (e.g. with a spoon), but most people do that anyway because it'll be a pain to hand wash if you don't.
I don't have a dishwasher. I don't know anyone who has a dishwasher. Heck, I don't even think I've ever seen a dishwasher on any appliance store I've been to. Our apartment was built with space for a dishwasher and a clothes dryer, and I've always wondered why, because nobody I know personally has either of these appliances.
I might imagine a Christian or Muslim (fundamentalist) society looking down on appliances that do 'womens work'. And I've spent some months in southafrican townships. Such places have no dishwashers. Or did not have them.
I'm not trying to doxx you, or attempting to get people to post prviate info, but I would love to know your demographic.
Southeastern Brazil, in the Rio de Janeiro metropolitan area.
Meanwhile there are no equivalent places where you can take your dirty dishes to wash.
Considering that "throwaway packaging and cutlery" is a common solution to this: Order food, the packaging, plastic forks and spoons and napkins included. And seeing that a large demography is trying to live off "less waste", I do see an opportunity here.
A community "diswashing service". It would need solid infra and tech to not immediately turn into a rat-infested-mould-party, but I truly think that a place where e.g. you drop your dishes, cutlery, glasses etc, and can pick them up in the evening filled with a meal, is an opportunity. For example.
Which reminds me, wasn’t room service once a perk you could get in some apartment buildings? You don’t often find dumbwaiters in apartment buildings any more, for example. I would pay for that if it were available.
That's extremely condescending, not to mention plain wrong.
I know from experience that the statement holds true for several Christian fundamentalists and Jewish fundamentalist (not all!)communities. I have only read (but not from experience) about how women (especially those of other sects) were treated in the Chalifate from ISIS and how their "work" was deemed slave-worthy.
It's still outright wrong. Your experience with Christian and Jewish fundamentalists does not apply to Muslims. Please stop with spreading incorrect information about Muslims.
ISIS is a fringe group whom we don't give any semblance of credence to, and even they would not look down at washing machines "because it's a woman's job". This is plain ridiculous.
The secular West continues to group all religions together and marks any religious person as some sort of sheep.
Also, let's stop with this "identifies as" business. A person is either a Muslim or a non-Muslim. A person who does not believe in God cannot "identify as" a Muslim for example. That being said, the religion is open to anyone who wants to accept it.
As long as you accept it is their right to define that, and not you!
One might use "identifies as" to avoid the classic response of "ah but x was not really a y because z". Someone who believes themselves to be a Muslim simply is. You can call them a bad Muslim, ignorant, misinformed, misguided etc. But they are what they believe themselves to be.
(I am entirely sympathetic to your original frustration. GP was applying very odd assumptions)
That's not how Islam works. That's how it was able to survive against the torrent of corruption that affected Christianity and Judaism. The Quran and Hadiths are preserved so that we're easily able to consult them to determine who is and isn't a Muslim. Note that this isn't gate keeping, Islam is wide open for anyone to accept. However, the tenants are strict enough to survive the passage of time.
> You can call them a bad Muslim, ignorant, misinformed, misguided etc. But they are what they believe themselves to be.
This already happens. I'm talking about the extremes that the secular West has adopted, whereby people can inherit their parents religion, even if they don't believe in it.
And yet those poor benighted souls believe they follow those very same texts that you say would reject them. How is that possible? Because their religious leaders interpret the texts differently, emphasise different aspects.
For a Christian equivalent. The Catholic church can excommunicate someone. They will no longer be Catholic but there is no power that can stop a person who believes themselves to be Christian from being termed that. "Normal" Christians might prefer not to be associated with various people, can call sects Extremist or Fundamentalist etc. We simply cannot claim they are Not Christians.
You'd be surprised to find that said "religious leaders" you're referring to don't know the basics of exegesis. This applies to ISIS for example. I invite you to read the many Hadiths about Khawarij, those immediately reject your premise that all interpretations are equally valid. They're not.
It is, also much much easier to identify a race condition with print debugging than a debugger.
Debuggers help immensely if you can reproduce the bug reliably and in relatively decent time, but if you can't logging/print debugging is your only hope, since no one is going to run their code in production with RR attached.
You think you're here? Let's put a few print statements in there to confirm your execution path is actually what you think it is, and those values are what you expect them to be.
> You think you're here? Let's put a few print statements in there to confirm your execution path is actually what you think it is, and those values are what you expect them to be.
You could just put a breakpoint there, look at as many values as you want, and not have to re-run it if you think of another.
To me that's the key difference. If I know what I'm looking for, prints are great. If I don't know what I'm looking for, breakpoints are a must.
Still though, I reach for print debugging first.
But in a normal environment ? or using C++ ? It's generally quicker (though much dirtier) to find simple bugs using print statements to track down assumptions ; when you're in the weeds with a complicated problem or need to track down a crash, then yes, bring out the big guns. But for normal development, I don't think it's completely necessary.
Eg: tracking the values of a std::vector isn't obvious, since there is std:begin and std::end iterators in the vector namespace and you have to manually dereference the whole range in the debugger.
I used to print-debug a lot, but since I started working with a stack where I use an IntelliJ derivative - that is to say, where the debugger actually works - and I'm never ever going back. It's better by a huge margin.
Have looked at, say, the Python debugger as used by PyCharm and thought "well this seems like a bludgeon, I only need to see how <variable> isn't working as expected" and bung in various print() statements in and around the Thing concerned, sprinkling them wider and wider if the bug isn't localised to a function or even the class it's in - I may have supplied naff information to a class when instantiating it, for example.
Debuggers are simply too much of a complicated bludgeon. I can understand a need for debuggers for machine code/assembly programming, or deep system programming when you're hitting the metal, or even low level languages like C.
I've yet to find a need for a debugger for high level languages like Python. Perhaps some day. Until then, I'll plough the time which would be required to Get To Know the debugger, into writing code.
Whenever I try to print debug, I end up missing the mark on my first print statement(s).
If I just set a breakpoint, I can jump back in the stack, or step forwards, so I can easily find the real place the error happens.
Of course I sometimes still print debug - when using languages with bad debuggers, languages where I don't know the tooling, languages with horrible control flow (javascript callbacks), or constrained environments, printing/logging is sometimes the best you can do. But that doesn't mean it's the best there is.
I put "if (condition): import pdb; pdb.set_trace()" all the time and almost never use "print".
So, clearly I don't love "print", I just hate debuggers that are annoying to set up.
I suspect that if other languages had such a convenient "drop me into the debugger" statement, more people would use debuggers.
1. Ability to prevent wasting resources on a system that does not need to be debugged. 2. Retain most print debug branches that were useful. 3. See how the internals are functioning on a live system with the issue that has yet to having a lab level proof.
It's even worse if it's multithreaded as well. If you hit a breakpoint, one thread stops and the others keep going, but they keep going in an environment where one thread is effectively dead (the one at the breakpoint). Everything is invalid from then on, in all threads. So breakpoint debugging is certain to be doomed.
Print debugging is not doomed, but it's challenging. The problem is that the time to print out the message can change the timing, and therefore change what happens. It's not certain to do so, but it's something you always have to keep in the back of your mind: "What if this changed the sequence? What if what I'm seeing in the printouts isn't what I would see if the printouts weren't there?"
I tend to use print debugs in two cases. One is when I'm pretty sure I know what the problem is, and it seems trivial. It's often faster to just confirm & fix that way. The other is when I am not even sure where to start, so I put a few canary prints here and there based on hunches until I start to narrow in mentally on what the issue might be.
Beyond that, it's a proper debugger.
No, it's quite possible to have a screwdriver in your pencil holder and turn it manually at the moment you need it.
"Low tech", yet quick and precise and no batteries to charge.
I clicked on this about to type exactly that phrase.
Here's why: the CORE PROBLEM of debugging has nothing to do with process, it's the need for the human developer to understand the operation of machine code when faced with a specific set of runtime conditions.
You can't debug by just reading code, because the bug only shows up at runtime. Most people agree with this.
Likewise, you can't debug just by WATCHING a program, for the same reason. You need to be reasoning about the operation of the abstracted code at the same time. And almost always you need to be inspecting and reasoning about the internal states driving the behavior.
Debugging tools, as nice as they might be, can only do the "watching" part. They are a fine way to do it, but you still need to be making the decision, based on careful analysis of the code, on which elements to watch.
So... given that you're staring at code in your editor and thinking about how to extract the needed internal state to validate your predictions... why not just print it?
That is: print/console debugging isn't a hard task you "have to do" in the absence of a fancy debugger, it's exactly the task that you must do if you're going to solve the problem. The fancy debugger, at best, is only saving your the time taken to TYPE the printf() expression. And in practice even that tends to parallelize with the analysis in my experience, as you realize things while typing out their expression.
https://pernos.co/about/expressions/
Notice in this demo how he clicked on a line number, and got back a list of all the times in the program where that line was executed. Not just a single breakpoint with the option to continue, but all of them. Then he queries for the value of a variable at all of those points, and gets all the values printed out simultaneously, as if there had been a printf at that line. Then he filters the list to show only calls made on a specific object, as if the printf had an if statement around it. All without needing to rerun the program, or even be on the same computer as where the recording was captured.
This is what the author means when he says that all debugging tools can be _better_ than printf debugging, if only we put some more time into improving our tools.
By all means, use fancy tools like replay debuggers (of which Pernosco is only one example) if they make you happy. I'm just saying that (1) there is a hard cap on the value that tooling can provide, because the real limitation is between our ears. And so (2) I'm going to spend the limited learning bandwidth between my ears on things like genuine new technology and not better tools that might provide a mildly incremental improvement to tasks I do already.
For example, I participated in a project recently to extend Flex so that it could generate both Rust and Go code, in addition to C. Pernosco was an invaluable aid here. I was able to get Flex to generate a C program and also a Rust program, then compare the execution of those two programs. I recorded them both, and opened them each in their own window. Keeping them side–by–side, I could probe the innards of both programs simultaneously. Since both programs were supposed to implement the same state machine (and thus recognize the same list of tokens), it was easy to find places where the Rust version went off the rails. I was able to do in hours what would have taken days with any other method, if I had succeeded at all. The behavior of that state machine is not at all obvious, and I might never have unraveled it without Pernosco.
See... I think that's just wrong. Here's how I'd do that (knowing nothing about flex but having written similar state machine generators for non-lexing problems):
+ augment the generated code to dump the sequence of state transitions and tokens to stderr or whatever
+ run it in both environments
+ diff the output
+ Go back and add some metadata like __FILE__/__LINE__ or whatever as needed to find the bad data or state
+ Repeat as needed
But I think that you’re underestimating the time that it would take you. Sure, if you knew ahead of time what you would need to log, it might take an hour or even less. In practice though you’re going to end up going through that loop dozens of times, adding some logging to both sets of source code (or removing something that turned out to be useless or confusing), rebuilding, rerunning the test case, and diffing the outputs.
Much better to record everything, and I do mean everything, using rr. Then you can add and remove data from your queries until you understand the bug (or rather, one of the handful of bugs that all existed at the same time) and how to fix it. No changes to the source code needed, no need to recompile them or rerun the test case.
However, I am perfectly willing to suppose that you could do in hours what would have taken me days. There are a lot of people in the world, and some of them are bound to be better engineers than I am. In that case, I suspect that using Pernosco, you could have done it in minutes instead of hours. I think it’s safe enough to say that gives me a straight 5× improvement to my abilities, and it would do the same for you.
Incidentally, roc has talked about adding automatic diffing of recordings to Pernosco. This would be diffing between multiple runs of the same program, rather than between two related programs that happen to be trying to accomplish the same task, but I can well imagine how much time it will save. Imagine those intermittent tests that fail one time out of a hundred. Currently you can rerun the test until you manage to capture the failure in a recording, and that helps to debug the problem quite a lot. But then imagine comparing a successful run to the failed run, so that you know the bug is in one of the differences between them. It’ll give anyone superpowers.
Of course there is some "cap" on the value tooling can provide, but whatever it is, for some people at least it's close to the total time they spend debugging.
Bullshit. I regularly find bugs in code review, just by reading code. I think this is true of most experienced programmers.
So yeah, in the text above I'd view "code review" and "static analysis" as distinct tasks from "debugging". And I'm only talking about the latter.
https://tenderlovemaking.com/2016/02/05/i-am-a-puts-debugger...
The reality is that you're not going to attach a debugger to a production service. If you are doing things correctly and have separate environments for test/integ/prod you are not going to attach a debugger. Even if you can attach a debugger, the odds of you catching something are pretty small (not talking about trivial things).
So you should: 1) instrument your code and have the proper debug/verbosity log levels for things 2) have a trivial way of altering the log levels 3) have a way to fetch the logs and diagnose what is going on solely based on the logs (and hopefully with something as simple as grep).
This is the way. There are times and places when debuggers can be useful, but in most situations I see them as a crutch that adds more friction instead of accelerating solving the problem
> Many people prefer print debugging over interactive debugging tools.
Then explains that eventually developers come around to debugging. He explains all the advantages debuggers have while simultaneously touting his own tools. Then he blames adoption on the lack of engineering effort put into overcoming teething problems and calls for platform vendors to better support debuggers.
I feel like the author is living in a vacuum and has no idea how most developers work. Watching his demo for Pernosco makes my eyes bleed.
The steps he takes for debugging a crashing Python App are:
1) Reproduce bug and capture with rr
2) Transmit capture to Pernosco
3) Wait Minutes for an Email to arrive.
4) Follow link in Email to Web based Debugger
5) Review Core Call Stack and Assembly for your Python App.
By the time the author got to step 4 of his Pernosco demo, most developers using Print Debugging would have fixed their bug and been on to the next one.
This, in particular, is why print debugging won't go away.
I would speculate that most developers, once they identify a bug in a downstream package, would instead either go online to find consensus around the bug and report it to the maintainer of that package, or just find an alternative solution.
Print debugging is awesome and good. It allows one to explore flow and data. It allows intuitive, multi-pronged examination of a program. It can easily be combined with all the tools the programming language offers. It is incredibly easy to use. It is the same 'UI' everywhere – and it is available – in all languages.
I rarely see something this bad make it onto the main page of Hacker News.
...except he did make some of those magical tools. This blog post is an advertisement for them.
> Print debugging is awesome and good. It allows one to explore flow and data.
It's the most primitive way to do that, which just happens to be "good enough" in many cases. This hinders the improvement of existing tools. This is also true for debuggers, they have been stagnant for decades. It's probably part of the same trend that pushed UNIX to domination despite being so primitive: It is cheap, portable and already ubiquitous.
I’m not sure “walking should go away” but it is not great when people have to walk for hours to go a few miles, in a universe where bicycles also exist.
So basically, this hypothetical forum is not on the internet.
It's perfectly fine for you to want a step-based debugger. In fact, if there's anything not fine is that you should probably want more than that, but don't even get to see better tools around. It's great that somebody took the time to develop better tools and is trying to sell it... But, let's stop attacking print, because it's a perfectly fine tool.
Log or printf debugging is like a shovel, RR is like a steam shovel, and Pernosco is almost like the entire railroad logistics system set up by John Stevens for digging out the Panama canal.
He completely rebuilt the rail system so that trains were moving past the shovels in the cuts continuously. Each shovel would simply unload dirt on whatever train car was closest. The trains in the cut moved slowly and never stopped, so there was always a car available. At the fill sites where the dirt was unloaded, he built what was effectively a huge plow system that would scrape the dirt off of the flat–bed cars all in one go (The French just had hundreds of men climb into the cars with shovels to dig the dirt out of the cars, effectively handling every shovel–full twice). The result was like a continuous conveyor belt over 50 miles long. He also built towns, stores, machine shops, water treatment and sewage systems for the towns at either end of the canal, and allowed Gorgas to virtually eliminate mosquitoes from the canal zone.
Pernosco is more like the entire logistics system than the steam shovel, though there are still a few rough edges and some features that I wish it had.
Sometimes you don't need the entire railroad logistics system if you're only removing a single shrub.
- Works in any programming language.
- Works in programming languages that aren't easy to debug with gdb out of the box without setup.
- Works if you're trying to figure out why a problem is occurring without a core dump
- Other developers understand how it works -- I find that there's no reasonable expectation that a developer knows how to use gdb.
- Works on systems you only have log access to -- true for production systems that have sensitive customer data.
So instead of debugging in two different ways, I've submitted myself to print statements -- tragedy of the commons I guess.
RR emulates pthreads in userspace, converting your multithreaded programs to run in a single thread. It does context switching between your threads much as the OS would if you had only one CPU core, but there is no true concurrency. This can hide some problems that are the result of race conditions, making them never show up in any of your recordings. If the problem doesn’t show up in any recordings, then RR isn’t going to help you debug them.
However, RR does have a mitigation for this, called chaos mode. In chaos mode it doesn’t schedule your threads fairly. It randomly starves some threads, not allowing them to be scheduled, or interrupts them before their usual time slice is up. This can make it easier to capture race conditions in a recording, though it isn’t perfect. You should give it a try some time.
For me, the biggest hurdle to using a debugger is having to stop and re-learn how to use it. I reach for print sometimes only because I can’t remember the command line for the python debugger off the top of my head, or I can’t remember how to inspect a hex value in gdb. Yes it’s only a quick Google search away, but I can try the print without searching Google. Perhaps irrationally, having a long list of good debuggers I’ve never used like rr and Pernosco and others actually semi-consciously causes me to want to use them less because of the time investment required to figure them out (and repeatedly, like it has in the past with other debuggers.) There are hundreds of tools that I’m not using to their full potential for my job.
Anyway, it’s silly to pretend we can’t have both. I land heavily on the side of using a good debugger whenever I can, and yet I still use prints all the time for quick & easy debugging. It’s silly to suggest that print debugging could ever go away; it can’t unless you want to remove print statements.
One thing both this article and the one on print debugging the other day completely fail to address is debugging embedded systems and GPUs, where both types of debugging are much harder. Try debugging your shader on ShaderToy.com - neither method is available.
It just works, and it works everywhere. That's why people use it.
Surely, if your work involves working in a certain stack permanently with a certain set of tools, definitely a dedicated debugger could improve your debugging. But for people who work with a large array of technologies and languages, especially senior developers who have to switch back and forth in between stacks - especially if they occupy a lead or man role in their org - that's not feasible. But print just works.
Additionally print debug is the program telling you how things are without something else interacting with it. Leave aside not having to go through creating breakpoints and then going through them slowly one by one (god).
You just place a print in a suspected location and run the program hands off. It doesnt trigger. Ok. You go to the other half of the potentially affected code. Put the print in there. Ha! it triggers. Then you repeat the process again, closing down on affected bit fast. Its a binary search pattern, and its fast. With debugger you have to create a mini program (thought-out breakpoints) to go through the actual execution of the code from the start.
Compare this to debuggers, you have to learn the special syntax of the debugger, which is usually very different to the host language, and only then can you start doing some actual debugging.
> If the software industry took debugging seriously — instead of just grumbling about the tools and reverting to print debugging (or, at best, building a polished implementation of the features debuggers have had since the 1980s) — and invested accordingly we could make enormous strides, and not many people would feel the need to resort to print debugging.
This doesn't make any sense, people don't "revert" or "resort" to print debugging, they use print debugging because it is so much simpler compared to debuggers. Print debugging is already as simple as it gets, you can't cut any complexity by adding a new tool.
In VS Code, you get the basic visual debugger UI that has been around for decades, to observe variables, set breakpoints, move about the stack, and step through the code. This works for pretty much all major languages. It's dead simple and easy to learn. It even works over SSH.
Now, if you insist on doing everything through a terminal, nothing will save you from a miserable debugging experience.
You said debuggers require learning "some special syntax", which is not true. This led me to believe that you weren't aware of things I wrote.
> VS Code's debugger is nice, but can you seamlessly plug in rr to get time travel features?
I haven't tried that, but apparently it should work:
https://github.com/rr-debugger/rr/wiki/Using-rr-in-an-IDE
> What about Pernosco?
The way I understand what Pernosco is (a recording), I would expect that to work as well.
I think you misunderstood my point, there are plenty of good GUI debuggers out there. But they don't support all environments. Some IDEs only support C/C++ or Java or something else. The point is that their support is nowhere near as good as print debugging. From webdev to kernel development the API is nearly the same. Even in VS Code sometimes you have to fall back to the terminal to deal with weird environments. It is never really as "seamless" as print debugging.
IMO, there is a big exception here when you want to understand the shape of data in a duck typed language. In that case, dumping an entire object can be useful just to see the shape of you data. This is something you could glean from reading but it generally doesn't change from one run to the next and is frequently left undocumented.
Is that verifiable true?
So, I'd say that when you use a debugger in tandem with a bunch of other good tools then the bad habits that the debugger encourages just matter less. There are fewer classes of bugs that will make it through verification and those other tools force you to understand the code at least well enough that you can pass their checks. However, you still need to read to generalize and fix those remaining bugs properly, and the debugger will push you toward a fast single point resolution rather than a slow but general resolution which is what you'd get from careful reading and thinking.
The big argument against this is I think is if you're working with something where a large number of bugs is acceptable. Like, if you're hacking together a demo/prototype and you just need X to work then a debugger might be the way to go: Just ignore the other bugs because who cares? Or if you're working with some legacy software that's generally known to be buggy and you just need to fix some specific critical use case quickly.
It's also just hard. Like you said, being able to read complex code and then systematically coax the bugs out of it with your brain and eyes definitely takes a lot of practice and the more that that can be supplemented by automation and good habits, the better.
And no core dumps are not a "solves everything solution" either. A) because dumping the entire 1500 TB of memory to disk takes a lot of time and B) because bugs can trash the stack, leading to misleading stack traces.
Inside significantly complex programs, plopping a breakpoint and then climbing the recorded stack frames gives you a snapshot of the program at all calls all the way up the stack. It lets me find information in a matter of moments instead of minutes. You can run commands right there in the current moment of the program you paused on, and you can even change values to what you think they should be and let it run to make sure your assumption is correct and your fix will actually work.
Instead of running the program ten times in a row to debug, I do all that in one go, and I can let my curiosity run wild as I poke and prod the program as I see fit.
For programs that take a while to run during development, it's not even feasible to print debug. If you're waiting 1-5 minutes to run the program, and it takes you 5 goes, you've blown nearly half an hour twiddling your thumbs.
Quite frankly if you were on my team and not using decent debuggers to troubleshoot, I'd have a chat. You and I are both on the clock, use your tools and get it done!
Caveat: Some debuggers suck, and print debugging is less painful, but still painful.
And there is also classical logic/algorithms problems: your program did not crash, but after 50 iterations of the main loop, the result set is empty. With prints, you get it all on screen and stare at the intermediate progress for a few minutes. With debugger, you can set a breakpoint... and then you have to hit "continue" 50 times, trying to remember what the values were 20 steps ago.
For your second point, logic/algorithmic problems are when the debugger shines the most. If you don't know where the problem is, you can put a breakpoint somewhere you know is after the problem then climb back up the recorded stack frames to find the issue. If the issue is in a loop, you can put in a conditional watcher to only break when your condition is met, or just set a breakpoint one line after the loop and inspect the loop result right there.
The most useful part of debugging is the ability to rapidly excercise your "aha" moments. When you print debug, you see the issue and go "Aha!" now you have to go verify with another print. If I see the issue in a debugger, I go "Aha!" and then immediately get to look at the variable or variables I've discovered is part of the problem.
I want to make sure it's clear that I don't think print debugging is useless, it's got it's moments of usefulness. I just don't understand how someone would not want to take advantage of an advanced debugger when it's available. They are so powerful, it would be like watching a mechanic use a pair of pliers when they have access to an impact gun.
And instead of putting a breakpoint and then hitting continue 50 times, you query the Pernosco database for all executions of that line, and it shows them to you as a list. You can then have it evaluate expressions at those points in time, and it shows them in the same list. It’s fast too; it can evaluate your expression at hundreds of points in time faster than you can type them in.
> when I don't have access to a debugger
Why can't we get rid of print debugging?
Because we don't always have access to a debugger.
If you would do that inside a debugger, you'd have to make a note when you hit a break point. That's not helpful. It gets worse when you can't really halt the program because there's some kind of real-time effect in your problem.
Yes, you can tell debuggers to print messages instead of halting, but that's just like adding a print statement. IMO, it's even better, because when you make a change, your print statement will still be at the correct point, while your breakpoint might have shifted a line or two...
I haven't personally used it, but I think the debugger in the suggested article even keeps a queryable database of data for every recorded execution.
I hear you and others though, I wasn't personally suggesting print debugging should go away, I still use it here and there.
It's not always the case if you like the one or the other. With out of the ordinary issues or places where you can't afford to sacrifice performance but still would like to have some insight printing usually wins by a large margin.
We should be at a point of maturity in software tooling where that never happens, at least for commercial software companies. I don't much mind what anyone uses in the free time or on their solo SaaS, but as service providers we should be aiming to use mature and effective tooling.
I'm not sure what you mean about performance, since debugging is something you should be doing once off in a local development environment, it shouldn't impact performance for users. Even videogame engines can be debugged. If you've got an issue only happening in production, you should probably try and replicate the production environment so you can debug it locally instead of messing about in prod.
For me it's the opposite, a build-compile-run cycle takes 10 seconds, but loading symbols in gdb... I generally can go and make me a coffee, even with gdb index
Some reasons why using a debugger in prod was hard:
- knowing where to start. With logs, you get a picture of what has happened.
- not stopping threads
- not everybody can do it, but everybody can look at logs.
Wrote more about it here: https://henrikwarne.com/2014/01/01/finding-bugs-debugger-ver... -
For more serious debugging, sure, a proper debugger is the way to go.
For example, my Go microservices run in containers built like this:
FROM scratch
COPY --from=build-container /go/bin/the-binary /go/bin/the-binary
However, if I want to debug this, I need to use a different container that includes Go delve (the Go debugger) and any dependencies it relies on, possibly open some extra ports, deploy that container instead of the standard one etc.
Cortex-M developer here. Let me know when you support bare metal.
It also records several hardware performance counters that were added to Intel CPUs over a decade ago to ensure that it can stop the playback of the program at the correct points reliably. It could also do this purely by single–stepping the program, but that would be horribly slow.
For an embedded system, or for debugging the Linux kernel itself, this doesn’t work. The kernel doesn’t make syscalls, and your embedded system might not even have a kernel to make syscalls to. And your hardware probably doesn’t have the right performance counters; AMD cpus only got equivalent capabilities quite recently.
On the other hand, perhaps you could capture and record interrupts instead of syscalls. Maybe the performance counters on the Cortex–M cpu could be made to work for RR’s purposes (I’m just assuming it has some performance counters; I haven’t checked). But that would take a lot of work, and most people just want to be notified when someone else has completed the work.
J-TRACE is the thing you buy when it's 5 minutes to production and your hair is on fire and you can't figure out what the problem is. Which is why Segger prices it at $2,000.
At least if you can do it in hardware then it doesn’t have to be slow, even if the recorded state is still huge.
I would like a debugger that would let me set up some query about the evaluation inside the program, for example, find me all input data in this test run that caused the output of a given function/expression to be such and such (possibly another data set produced during the debugging). Find me what are the typical inputs to the function. And so on.
One of the ideas would be instead of having breakpoints set to a place of execution, they would be set to passed values or program state. This is much more sensible in the world of functional programming, and processing of large amounts of data.
I can imagine a system like that to replace print debugging for me, but until then.. I find it easier to print.
https://pernos.co/about/expressions/
Notice how the user clicked on a line number, and a window opened that shows every execution of that line. It’s not a breakpoint where you see only a single point in time; instead you see all the points in time. He then types in an expression, printing the value of the named variable at every single one of those points. Lastly, he types in a condition expression, narrowing the list to examine just the calls for a specific object.
I can see how replay debuggers work with programs like Firefox, where you start with a few megabytes of input data and have a working set of a few hundred megs -- but there are programs with much higher memory/io usage out there.
Something as simple as "download a few gigs of data from internet and calculate its hash" will require enough trace to record all data downloaded, which would make the overhead huge.
Still, disk space is cheap; I may have to delete that recording one day but there’s plenty more disk space available.
I found that it was also a reasonable estimate for Reposurgeon, which is written in Go and uses concurrency during some repository operations, and when doing garbage collection. Other parts cannot be parallelized at all, so it balances out. I never measured that very precisely though; it would take anywhere from seconds to minutes to run a test without RR (depending on the data I loaded, etc), and only somewhat longer with RR.
rr only logs sources of non-determinism, such as system calls (which includes I/O). Then all the deterministic state of the program is reconstructed at runtime using the recorded non-determinism.
So having lots of state doesn't necessarily affect trace size but you're right that lots of I/O will.
I work on servers that are 150 to 200ms away in terms of latency. They're impossible to do interactive debugging on them (well not impossible, but really unpleasant).
Debuggers are not latency aware, they suck at it, they assume the debugged process is close to the UI showing the tooling, close in terms of latency and bandwidth.
Granted, most of the debugging is local, but sometimes replicating the environment so you can reproduce the issue is impossible or at least really time consuming, so remote debugging is a must, and when that happens, print debugging tends to win.
My bias here is infrastructure, and you simply can't attach a debugger to a running production process to slow down customers.
At core, debuggers are an educational tool for new developers or inexperienced developers. Experienced developers will add metrics, assertions, and logs to the code which will make print debugging mostly redundant.
I don't "prefer" print debugging over interactive debugging tools.
I need something that works reliably, repeatably and with as much difference from the production build as possible.
gdb fails for me in all of these ways.
I would prefer a working debugger, yes.
The first thing I do after `git clone` and getting the build working is spend a couple minutes trying to get a debugger working, preferably using the IDE debugger. Usually it's pretty simple and it works out of the box. Sometimes it requires fiddling a bit with debug ports and cli arguments (legacy Tomcat applications) or making sure sourcemaps are available (web frontend dev).
If I can't get it working within 15 minutes, printf it is. But in most higher level projects it's usually a trivial initial time investment that pays off in the long run.
1. Sometimes the information you can print about an object is not complete due to console memory constraints, depth of the information, dereferencing, formatting, etc. So you will have to come back again and print more ranges of that information, and format the data to understand it.
2. Logging takes "a lot" time if you are working on high performace algorithms. Then it will be dangerous to forget removing a print.
3. Prints can leak security information.
4. In web browser consoles you get a live print of objects, not snapshots.
That said, print is my favorite debugging method, but I constantly search and remove them, specially on release.
Some of the Mozilla folks did livestreams of their development work on Firefox, including debugging sessions. I haven’t watched any lately, but you might search them out. If they kept it up then they might be using RR now, instead of just GDB.
Here’s a demo video for Pernosco: https://robert.ocallahan.org/2019/10/pernosco-demo-video.htm...
Pernosco is based on RR, but this probably won’t teach you anything about using RR, since the UI is so different. But it was a memorable one because Brendan Gregg produced a really good tutorial for GDB, where he debugged a real problem in a real program (instead of a made–up problem in a program written just to have a made–up problem inserted into it). The Pernosco demo debugs the same problem in the same program, so comparing the two can be quite easy. I would like to see someone the same type of tutorial for RR, and for other debuggers that people like.
Using a debugger I find myself repeating the same steps over and over again that I would have done/written once via print debugging.
That and once my system is deployed and I need to debug issues in production I'm relying on logs which are effectively print statements. So why would I want my dev debug and production debug flows to diverge? Doing print debugging gives me experience in what sort of information is useful to log as well as what is loggable.
This is the only thing that visual studio does better than any IDE on the market... zero-config debug, even some mature IDE's like jetbrains ones still to this day don't have zero-config debug, it always take some external and obscure configuration to get things right.
You can even edit your call stack manually, save the stack manually and restart it later on any machine.
RR probably is pretty similar for nodeJS but I haven’t had a chance to use it. Is it a game changer?
For instance, from a "time travelling debugger" I would expect some sort of graphical time line view, where I can immediately see where on the timeline I am currently, and jump back and forth while inspecting program state and memory in UI panels.
A good UI is the most important part of a debugger, and the area with the most potential for improvement (Visual Studio is generally praised as the best debugger around, but when you think about it, the UI is archaic, yet it's the best we have).
Without a good debugging UI and intuitive workflows, it's no wonder people are falling back to printf-debugging.
I can't quite tell if the author is arguing against DEBUG/TRACE logging which is the gold standard in debugging, particularly in low reproducibility and multi-threaded/actor-based systems or systems with many 3rd party dependencies.
Several systems cannot even use debugging. Such as realtime, kernel or embedded. The only tool they have is debug tracing. Looking at logs. I've worked years only looking at logs, tons of logs from tons of connected real-time systems. Writing tons of log analysis tools.
I love debuggers, but debuggers are a rare luxury to have. In many cases we even added embedded shells (read-eval-print loops, lua, picoc, ...) into embedded systems to query its state. Similar to a debugger, but also completely different.
You should also not forget LED debugging. Sometimes you only have two LED's.
For debuggers it's probably the same: if they make the assumption that you need less than complete freedom in instrumenting the code, you will need print and if they don't assume that, they likely are harder to use than the language itself.
Better to use a language which has these included, is there any?
I hope you’ll give RR a try; it’s been fantastic for me. (rr-project.org)
What I've been thinking about is a code generation tool where I can specify the variable(s) I want to watch and it takes the original code as input and produces code with traces everywhere that variable is referenced or modified. If you're going to do print debugging then let's go all the way!
So I use print debugging exclusively.
If you make any wrong assumptions about freezing a program at a certain spot with a debugger ... I’m not sure, I guess I’m super biased about the simplicity of a good sharp kitchen knife.
The difference is that people have proven to be extremely quick to spend time and money on installing plumbing, but these days hardly anyone spends much time improving their editor or other tools. Some companies, like Microsoft, are rich enough to employ dozens or hundreds of engineers on VSCode, and then give it away for free in order to build good–will with a whole generation of programmers. I’m given to understand that Google has teams devoted to developer experience, who spend their time sharping the tools used by Googlers so that most Googlers won’t need to. But most of us don’t spend much time improving our tools, and we mostly can’t buy better ones.
It may be presumptuous, but I agree with Roc. If we spend more time on our tools, then print debugging _will_ go away. Not because we’ve forced you to use terrible tools, but because the tools will be so much better than print debugging that you’ll voluntarily start using them.
I'm not saying that "debuggers should go away" because I'm not going to presume that I know what works in someone elses situation, but it seems like this article is actually doing the exact same thing to me.
Everyone on that "side" seems to be more emphatic that "print debugging should go away" (the very title), yet for the most part, I'm not seeing print debugger types demanding that interactive debuggers should go away.
So there's a disparity in those two attitudes, that implies some advantages on one side or the other, wouldn't it be more productive to list the advantages and let the merits of each approach win out in each situation we programmers find ourselves in?
Not to mention my "vendor paranoia", I mean the guy already is making my agent orange act up by almost titling his article "the way mousepilot works should go away," that isn't exactly the sales pitch thats gonna win me over.
This is not at all what he is saying. Pernosco makes print debugging even better, faster, more precise, and with less room for errors. It also enhances every other aspect of the debugger, by making them omniscient as well, but nobody will force you to use those features. No one will make you click on a value to trace backwards in the program to find out where that value came from (https://pernos.co/about/dataflow/), or to debug multiple processes that were running at the same time (https://pernos.co/about/multiprocess/). You don’t have to use these advanced capabilities if you don’t want to. Perhaps you’ll never care who the callees of a function are (https://pernos.co/about/callees/). Maybe your programs run purely in the terminal, so you never need to click on a pixel to find out who drew it (https://pernos.co/about/screenshots/).
He’s saying that he thinks that once you have the option to do more than just print debugging, you will. Once the debuggers for your favorite languages support these features, you’ll gradually use them more and more, until you hardly ever do “just” print debugging any more. But to get there, someone has to put in the time and effort to make the debuggers for those languages good, not merely acceptable. Or Pernosco has to be extended so that it recognizes the interpreters or runtimes for those languages, so that it can extract the debug info necessary to let you debug your program instead of the interpreter. It can already do this with Chrome and node.js, which both use v8 to run Javascript. (https://pernos.co/about/javascript/)
When Python developers see that, they should sit up and wonder what they can do to have that feature for themselves. They can implement it themselves in their own debuggers, or they can extend GDB to provide extra support for the Python interpreter, so that GDB can show the source code of the Python program instead of the source code of the interpreter. Maybe they could help extend Pernosco too. Same with Erlang, and Scheme, and Go, and Java, and so on.
> doesn't that sort of require me to believe print debugging is the same as carrying water from a well?
Sure. And I get that it’s not as obvious as comparing buckets to pipes. Once you see how a pipe works, and a faucet, it’s pretty hard to deny that they have advantages over buckets in nearly all situations. It’s also hard to argue that buckets have any advantages over pipes at all.
But with debugging it’s harder, because the PHP debuggers are awful, and the Python debuggers are barely adequate, and the Bash debugger barely exists, GDB’s command set is about as tidy as kudzu smothering a stand of trees, and so on. It’s easy to argue that these tools are inadequate, and easy to dismiss all debuggers as a result.
But the sad state of the debuggers for many languages is due to primarily to neglect, not lack of potential. Even the fact that new languages need to write their own debuggers is sad commentary on the human condition.
So yes, print debugging needs to go away, just like poverty and death need to go away. We can solve the problems, we just have to get up and do it.
Ok, but I am not going to set up plumbing everywhere I go when a water bottle does the trick.
>If we spend more time on our tools, then print debugging will go away.
No matter how much time we spend on plumbing will make the water bottle go away.
This one has been around long before computers existed and the chances of it going away are low to none.
Note: Heck, even Java will probably stick around for quite a while ¯\_(ツ)_/¯
That's print debugging, and it's not going anywhere as long as unit tests are around.
But I've had debugger crashes or weirdness where they stop working in every language and debugger option I've ever used (gdb, lldb, MS debugger, JVM, Rust, V8, etc).
On a realated note, the number of times I've had gdb crash on arm is so ridiculous that I don't really bother with it unless I absolutely have to.
Embedded debugging is a pile of woe as well, in my experience. On the current system I'm working on, with 4 different chips using 2 different architectures, 3-4 different JTAG adaptors and different vendors gdb integrations, none of the debugging interfaces work reliably. If you can get the software to start running under the debugger, usually the first breakpoint will get hit correctly. If you're lucky it'll even continue running after you restart it from that breakpoint. The next breakpoint is even less likely to work, let along actually being able to trace out the path of some code. In theory on one of the architectures the interface understands freertos threads, but that just reduces the odds of a successfull breakpoint even lower (and you can forget about support on the other ones).
I don’t know if I agree with you there. I suppose you did waste your time, since the bug turned out to already be fixed, but every crash has to be investigated by somebody. Otherwise, the wouldn’t get fixed. If they don’t get fixed, they simply waste everyone’s time, over and over again.
The sad statement above was that the commenter saw a crash in a tool, and gave up instead of investigating. Thank you for not giving up when you were in the same situation!
> Embedded debugging is a pile of woe as well
I’ve never done anything of note on embedded platforms, but it certainly sounds like it. You would think that you could pay the vendors for better tools, but it sounds like they all just do the bare minimum, or they just do the bare minimum for anyone who doesn’t pay through the nose.
I can't bill a customer for that time, and my boss won't be able to pay my salary if I spend weeks studying and fixing broken old debuggers instead of doing the work a customer pays for.
Either way, the debugging experience without something like rr sucks hard when it comes to heavily multi-threaded, asynchronous, event-driven systems. So even if gdb didn't crash, its usefulness is very limited compared to simple tracing.
Also, Undo works on ARM.
However, if you are logging the entire state of the program over time then you're probably better off using an omniscient replay debugger.
10 years later, I was still being spoiled while building Windows CE OS images. There you had KITL to debug (over serial, ethernet or USB) the OS in real time from the IDE.
When none of that worked, I purchased myself a Lauterbach probe (6K USD). The debugger IDE is the ugliest thing ever, but it's the most powerful tool around for embedded.
Some years later I entered into kernel (Linux) building for embedded. When asked the guru what's there to debug the kernel, the answer I got was "printf". Man, I laughed so hard until I saw his "I'm being serious" face. Then I get used to it, but I still hate it.
Again, some years later, I have to build Android images. Again, skimming through thousands of log messages thanks to dmesg and logcat... I still hate it. It seems archaic.
The better moments of my current work is going bare-metal, debugging with an IDE, with my 15 euro JTAG plus OpenOCD and Eclipse. I couldn't be happier.
In fact, for bare-metal embedded, there is no other way around. There is no printf. You can have it with semihosting or a serial port, but it's painful... it messes up with my timings. I need to debug using an IDE. And GDB from the command line... come on... it's 2021. It can be avoided.
Even when I started with Rust for embedded I wanted a full debugging IDE (which I didn't find yet). I want to see how things get arranged in memory (how does a slice or enum look in memory? and, is it possible no one checked this before?) side-to-side with generated assembler.
TL;DR, my opinion is that printf might be fine for some desktop development, but as a "I don't have any else to try" resource.
It's still unclear to me that learning an interactive debugger would save me any time at all.
Figuring out where I need to judiciously place print statements also helps me to figure out the code in front of me.
Is there anyone out there who is actually familiar with a debugger for the tool they're using and still prefer print debugging?
No?
That settles that then...
I for one use both print debugging and actual debuggers (I've used the C# VS debugger, windbg, Java debugging in Eclipse, NetBeans, and IntelliJ, CLI gdb and pdb, Emacs GUD for gdb, pdb and Go delve, Emacs edb and the CCL debugger from SLIME). Sometimes it's awesome to point a debugger at a core dump and fix a year-old memory leak, or just see windbg point out a deadlock involving 5 locks in 2 different processes. Other times it's awesome to interpret debug logs and figure out a how a flow in a customer setup got interleaved and produced a race condition two weeks ago, which is causing a crash today.
Of course a debugger can't replace the need for logs entirely!
The article is in fact claiming that a debugger can replace print debugging entirely, that's why I read your comment in this same vein.