Printf debugging is ok
polymonster.co.uk
polymonster.co.uk
This sounds so dumb but it works out to be equivalent to some very powerful debugger features. You don't need a magical debugger that lets you modify code on-the-fly while continuing the same debug session... just change the code and re-run the test. You don't need a magical record-replay debugger that lets you go "back in time"... just add a printf earlier in the control flow and re-run the test. You don't need a magical debugger that can break when a property is modified... just temporarily modify the property to have a setter function and printf in the setter.
Most importantly, though, this sort of debugging is performed using the same language and user interface I use all day to write code in the first place, so I don't have to spend time trying to remember how to do stuff in a debugger... it's just code.
BUT... this is all contingent on having fast-running automated tests that can reproduce your bugs. But you should have that anyway.
Absolutely! Running a commandline debugger adds additional context I have to keep in my head (syntax for all the commands, output format etc), that actively competes with context required to debug my code. Printfs work just fine without incurring that penalty, granted this argument applies less to IDE debuggers because their UX is usually intuitive.
Ideally, yes. But for many bugs, getting to a reproduction is already more than half the battle. And a debugger can help you with that.
Today I am working on a bug where a token expires after 2 hours and then we fail to request a new one instead we just keep on using the now expired one, which (of course) doesn’t work. I have a script to reproduce it but it takes 2 hours to run. It would be great if there was some configuration knob to turn down the expiry just for this test so we can reproduce it faster - but there isn’t because nobody thought of that.
Well, the hard part isn’t actually the code change (reasonably obvious) it is deploying my own copy of their service…
Or modify the expiry time locally / not use the one from the token at all.
Another thing I’ve found helpful, is to write out a “system state dump” (say in JSON) to a file whenever certain errors happen. Like we had a production system that was randomly hanging and running out of DB connections. So now whenever the DB connection pool is exhausted, it dumps a JSON file to S3 listing the status of every DB connection, including the stack dump of the thread that owns it, the HTTP request that thread is serving, the user account, etc. Once we did that, it went from “we don’t understand why this application is randomly falling over” to “oh this is the thing that is consistently triggering it”
When it writes a dump, it then starts a “lockout period” in which it won’t write any further dumps even if the error reoccurs. Don’t want to make a meltdown worse by getting bogged down endlessly writing out diagnostics.
Not my original idea, I got it from some Oracle products which have a similar feature (most notably Oracle Database)
Where may I read about particular workflows involving debuggers, e.g. gdb?
(Mainly for programs written in C.)
Most of my issues are related to issues with concurrency though, deadlocks and whatnot.
> they are often the 1 out of 100 runs kind
Can you share an example? In my whole career, I have only seen one or two of them, but most of my work is CRUD type of stuff, not really gaming or systems programming where such a thing might happen.You reuse connections with a connection pool, but you accidentally reuse connections with different privileges and scopes. As a result, sometimes you get to read some data you shouldn't read and sometimes you don't.
Or, concurrency bugs.
You don't properly serialize transactions and sometimes two transactions overlap in time leading to conflicts.
But even with a debugger, there's still loads of value of sitting up, moving from writing a bunch of single line statements all over, and to writing a real test harness ASAP to not have to rely on the debugger.
For any non-trivial problem, you'll often very quickly appreciate a properly formatted output stack, and that output will be shaped to the problem you are looking at. Very hard for an in-process debugger to have the answer for you there immediately.
Being serious about debugging can even get into things like writing actual new entrypoints (scripts with arguments and whatnot) and things like adding branches togglable through environment variables.
I think a lot of people's mindset in debugging is "if I walk the tightrope and put in _one more hack_ I'll find my answer" and it gets more and more precarious. Debugging is always a bit of an exercise of mental gymnastics, but if you practice being really good at printf debugging _and_ configuring program flow easily, you can be a lot less stressed out.
Like if you think your problem is going to take more than 20 minutes to debug, you probably should start writing a couple helper functions to get you on the right foot.
The main symptom was a non-deterministic crash in the middle of a 15-minute multi-threaded execution that should have been 100% deterministic. The debugger revealed that the contents of an array had been modified incorrectly, but stepping through the code prevented the crash, and it was not always the same array or the same position within that array. I suspected that the array writes were somehow dependent on a race, but placing a data breakpoint prevented the crash. So, I started dumping trace information. It was a rather silly game of adding more traces, running the 15-minute process 10 times to see if the overhead of producing the traces made the race disappear, and trying again.
The root cause was a "read, decompress and return a copy of data X from disk" method which was called with the 2023 assumption that a fresh copy would be returned every time, but was written with the 2018 optimization that if two threads asked for the same data "at the same time", the same copy could be returned to both to save on decompression time...
But I consider myself lucky that the issue could be reproduced on a local machine (arguably, one with 8 cores and 64GiB RAM) and not only on the 32 core, 256GiB RAM server. Having to work remotely on a server would have easily added another week of investigation.
Anecdote aside, it certainly doesn't hurt to be able to debug things without a debugger if it comes to that.
"Oh sure, lemme just set a breakpoint on this network service. Hm... Looks like my error is 'request timed out', how strange."
That having been said: there are some very clever solutions in cloud-land for "printf" debugging. (Edit: forgot this changed names) Snapshot Debugger (https://github.com/GoogleCloudPlatform/snapshot-debugger) can set up a system where some percentage of your instances are run in a breakpointed mode, and for some percentage of requests passing through the service, they can log relevant state. You can change what you're tracking in realtime by adding listeners in the source code view. Very slick.
Time travel debugging (https://en.wikipedia.org/wiki/Time_travel_debugging) can help with this because it separates "recording" (i.e. reproducing the bug) from "replaying" (i.e. debugging).
Breakpoints only need to be set in the replay phase, once you've captured a recording of the bug.
Also, will we ever move forward from these sort of discussions? Back when I was a mechanic no one argued about basics troubleshooting strategies. We just aimed to learn them and apply them all (as necessary).
You're correct, the real issue comes down to
1. making sure those printf statements don't wind up in prod, spilling potentially sensitive data or corrupting a data stream
2. making sure that non-printf tooling is built so that only printf debugging isn't used
We tend to get caught up in false dichotomies.
I’ve found before that sometimes I can’t see what’s wrong with the code on my screen but I can when I print it out. I think the printed page activates different regions of the brain compared to looking at a computer screen
Printing is obviously fine if you like doing that, but I've found there are lots of ways to shake yourself loose and more thoroughly review your own writing.
One time, I was looking for a set of keys which had been tossed over the aisle partition in a grocery story and promptly disappeared. We tore that place apart and still couldn't find them. So I was laying on my back on the aisle floor so I could see the search area with my head upside down, and there they were. They had bounced and were hanging from the bottom of one of the shelves.
When in doubt, do something weird.
Present them as options. "If you like X, you may also like Y." Leave it to the audience to discriminate when to apply the tool.
We need to have people continue the meme by missing the point of nitpicking nitpicked arguments. That’s a special thing here in HN.
Long story short, our game worked as long as the printfs we had were kept, we had macro to remove them (in "Release/Ship") but the game crashed.
The crash was due to side-effect of printf clearing some math errors.... So here you go!
one thing I think the "just do print debugging" folks miss is what a good teaching tool a visual debugger is: you can learn about what a call stack really is, step through conditionals, iterations, closures, etc and get a feel for how they really work
at some level being a good programmer means you can emulate the code in your head, and a good visual developer can really help new programmers develop that skill, especially if they aren't naturals
i emphasize the debugger in all the classes i teach for this reason
I think you meant "a good visual debugger". And I agree completely.
A visual debugger isn't just a tool for fixing bugs.
It's also a tool for understanding the code.
At my last job, the codebase was so complicated that you could spend hours scratching your head over what was going on in a function, especially how the code got to that function and what data the calling functions had that led to this point.
Of course you could add print or log statements, but then the question is what to print! And which calling functions needed more print statements.
With a visual debugger, I could just set a breakpoint in the confusing function, see all the data it had, and also move up the stack to see what data all the calling functions had.
There are cases where you need print debugging. I added one feature that worked perfectly locally and on a test server, but failed in a Jenkins job (ironically running the job on that same test server).
That was a case where I added print statements throughout the code, just to see how far it got when running locally vs. under Jenkins.
There are many ways to debug a problem. It is wise to be familiar with all of them and know what to use when.
If you don't know what a debugger does though that's something you should really get on ASAP. Likewise if you can't figure out how to get log messages out of your thing. Really all there is to it, figure out what you want to do after than and spend your time actually doing something productive instead of getting in a stupid holy war on the internet about it.
printf'ing effectively enforces a similar condition to `volatile` on the underlying memory segment when it is read.
One can encounter tersely written code that works perfectly with printf statements, but status bits never get "updated" (CPU cache purged) without the printf and hangs the program.
I work in a natural science, and use computing for numerical simulations and data analyses. For coding problems, a debugger is pretty handy. But for finding errors in the underlying mathematics, I tend to rely on a printf -> grep -> analysis chain. This might make files of several hundred Mb in size. The last part is generally done in R, because that lets me apply graphical and statistical methods to discover problematic conditions. Often these conditions do not crop up for quite a long time, and my task will be to find out just what I did wrong, with something like a problematic formulation of boundary condition that created an anomalous signal near an edge of a domain, but only once a signal had propagated from a forcing zone into that region of state space.
printf debugging is a symptom of poor tooling. It is like saying that driving in nails with a rock is fine. It works, but the truth is that if you are using a rock, that's probably because you don't have a good hammer. And if on every job site, there are seasoned workers banging rocks like cavemen, maybe hammer manufacturers should take notice.
So my debug builds often self document at runtime, and in production the printfs become comments.
Honestly I think attaching a debugger should be the first debugging tool you reach for. Sometimes printf debugging is required. Sometimes printf debugging is even better. But a debugger should always be the tool of first choice.
And if your setup makes it difficult to attach a debugger then you should re-evaluate your setup. I definitely jump through hoops to make sure that all the components can be run locally so I can debug. Sure you’ll have some “only in production” type bugs. But the more bugs you can trivially debug locally the better.
Of course I also primarily write C++ code. If you’re stuck with JavaScript you maybe the calculus is different. I wouldn’t know.
The typical modern dev loves to shit on Windows. But Visual Studio (the adult version, not VSCode) is still a best-in-class debugger. Xcode is bloated as hell but did help me last week. Linux has… poor bastards.
Also: Do also say that LLDB is a "dog shit trash debugger"?
Does Visual Studio do time traveling yet? https://www.replay.io/ is cross-platform.
WinDbg has a time traveling debugger.
replay.io does not support a single language or environment that I care about.
Sometimes a series of print-statements are better for helping you understand exactly when and how a bug occurs. This is particularly true in situations where the debugger is difficult or impossible to use (e.g., multi-threading). Of course, in that situation, logging may be better, but that's just glorified printf.
How are those cases debugged then? By enabling the debug symbols AND the optimizations and using the debugger, looking at the code and the disassembly side by side and trying to keep your sanity as the steps hop back and forth through the code. Telling yourself that the bug is real and it just cannot be reproduced easily because it depends on multiple factors + hardware states. Ah! I sometimes miss those kinds of bugs which make you question your reality.
It’s almost never the compiler. It’s almost never an error in the bare metal.
Almost.
When adding a print or a check - the compiler would choose different variables to store in the registers. They would still get overwritten by the faulty function but the bug would not be observed.
I agree that it's almost never the compiler's fault though - but sometimes its optimization choices make it harder to reproduce a bug.
Edit: The faulty function was a somewhat standard function, part of the SDK. This taught me that the standard functions are almost never faulty. Until they are :-)
> I know for some people this is often a terrible UX because of the performance of debug builds, so a prerequisite here is fast debug builds.
The reasons debug builds perform badly are kind of mixed, in my experience looking at other people's set ups:
Building without optimisations
It's fairly common to believe that debug builds have to be built with -O0 (no optimisations) but this isn't true (at least, not on the most common platforms out there). There's no need to build something that's too slow to be useful.
You can always add debug info by using -g (on gcc / clang). Use -g3 to get the maximum level. This is independent of optimisation.
You can build with any level of optimisation you want and the debugger will do its best to provide you a sensible interpretation - at higher optimisation levels this can give some unintuitive behaviours but, fundamentally, the debugger will still work.
Gcc provides the "-Og" optimisation level, which attempts to balance reasonably intuitive behaviour under the debugger with decent performance (clang also supports this but, last I checked, it's just an alias to -O1.
Doing a ton of self-checks
People often add a load of self-checking, stress testing behaviours and other things "I might want when looking for a bug" to their code and gate it on the NDEBUG macro.
The logic here is reasonable - you have a build that people use for debugging, so over time that build accumulates loads of behaviours that might help find a bug.
The trouble is, this can give you a build that's too slow / weird in its behaviours to actually be representative. And then it's no use for finding some of your bugs anymore!
I think it would be better here to have a separate "dimension" for self-checking (e.g. have a separate macro you define to activate it), rather than forcing "debug build" to mean so many things.
In VS Code, if you want to run debugger with arguments (especially for CLI programs), you have to put these arguments in launch.json and then run the debugger.
This is often tedious to do, because I usually have typed these arguments and tested in terminal before, and now I have to convert them into json format, which is annoying.
To make it worse, VS Code uses a separate window for debug console than your main terminal, so they don't share history/output.
So if I know what to look at already and don't really need a full debugger, I often just use print() temporally.
In theory, you can make conditional breakpoints very fast using an in-process agent. For GDB (for instance) this gives the ability to evaluate conditional breakpoints within the process itself, rather than switching back to GDB: https://sourceware.org/gdb/current/onlinedocs/gdb.html/In_00...
I've always found the GDB documentation to be a bit vague about how you set up the in-process agent but I found this: (see 20.3.4 "Tracepoints support in gdbserver") https://sourceware.org/gdb/current/onlinedocs/gdb.html/Serve...
When we implemented in-process evaluation of conditional breakpoints in UDB (https://undo.io/products/udb/) it made the software run about 3000x faster than bare GDB with a frequently-hit conditional breakpoint in place. In principle, with a setup that just uses GDB and in-process evaluation you should do even better.
Are you diligent enough to remove your sensitive logging/printf statements EVERY time, for the rest of your career? Or should you make a habit of doing things properly from day one?
Yes, when a linter is set up that fails when printf debugging is found
If we have no other option then sometimes we have to use non-ideal approaches, but I don't get the impulse to start saying that tooling/observability poverty is actually good.
That said, there are some contexts where this is reversed - printing something useful without affecting the debugged code may actually be more involved than, say, attaching a JTAG probe and stepping through with a debugger. Though sometimes both of those are a luxury you can't afford, so you better be able to manage without them anyway (and this may happen to you regardless of whether you're working on low-level hardware bring-up or some shiny new JavaScript framework).
Everytime you have to do a printf it's a slowdown, and you can't out-argue the fact that you have to type up to 20ish keystrokes and excite a number of brain neuron cells trying to remember what that printf syntax or variable name was. In comparison to a debugger that automatically prints out your call stack and local variables even without you having to prompt them.
The key insight is that printf() is a heavyweight operation ("What, you want to build a string? A human readable string? Okay, one second, lemme just pull in the locale library..."). If you're debugging something at the business-logic layer, it's probably fine.
If you're debugging a memory leak, calling a function that's going to make a deep-dive on the stack and move a lot of memory around is likely to obscure your bug.
shudder memories of my early days in kernel-level programming where using a printf used to just "fix" some bugs.
That came down to an uninitialised variable (which calling printf was helpfully initialising by using the stack).
As a result of that early experience, when I'm in the headspace of very low-level bugs, I find it helps to think of printf both as "this will tell me some variable values" and as a source of other clues: if it makes a weird value disappear or change then you might have uninitialised data on your stack, if it makes a flaky behaviour become stable (either disappear or become repeatable) then it's probably a race condition, etc.
A reference to James Mickens' "The Night Watch" feels appropriate here: https://www.usenix.org/system/files/1311_05-08_mickens.pdf
If you are worried about including <stdio.h> and potential side effects from that, you can use __builtin_printf instead.
When I say move memory around in this context, I mean do a lot of stack operations. You can leak from the stack too (drop a pointer from the stack without freeing the underlying heap memory it referenced), and it's harder to catch that if printf has come along and completely rewritten your unused stack memory as consequence of reporting on the state of your program.
That having been said, the point is that context matters and what you're debugging matters for the question of what tool to use. If you're operating in an interpreted language, you can probably trust that The interpreter is making it difficult to leak memory like that. On the other hand, interpreters have bugs too, and using a language that is interpreted instead of compiled machine code makes bugs in the execution layer unlikely, but not impossible...
The decimal separator is a bit of a problem for JSON generation, too. Some systems have snprintf_l, but it's not very widespread.
My print statements normally come when "I have no idea why this is breaking", and I start sanity checking the basics to make sure that things I previously thought were true, remain so.
Just recently I was doing something in C after a long time, and had something like this (simplified):
#include <stdio.h>
int main(){
int a = 0; // Input from elsewhere
switch(a){
case 0 :
printf("0\n");
break;
defult :
printf("?\n");
break;
}
return 0;
}
It was late at night, it compiled, so I knew it wasn't a grammar issue. But after testing with printf()'s I realised that the default case was never being hit and performing the core action. It turns out 'defult' highlights in my editor and compiles fine in GCC. Turns out that any word in that location compiles fine in GCC. Nasty!Anyway, the point was, my tired eyes did not easily grep the spelling mistake, hence the print statements.
Other than that, people should spend time learning the ins and outs of their debugging tools, like they do for the compiler switches and language details.
Additionally, when having the option to pick the programming language, it isn't only the syntax sugar of the grammar, or its semantics that matter, it is the whole ecosystem, including debugging tools.
Personally I rather have a great debugging experience than less characters per line of code.
That said, I was pleasantly surprised I was able to attach a debugger to that system. Some bugs really needed it.
[1]: https://andydote.co.uk/2024/11/24/print-debugging-tracing/
sadly there is no standard way to do this (c++ is reportedly getting one in '26: https://en.cppreference.com/w/cpp/utility/breakpoint), so you just need to use what your platform provides. here's a partial list: https://stackoverflow.com/a/49079078
e.g. __debugbreak in msvc, asm("int3") on x86[_64], raise(SIGTRAP) in posix.
Sure you can printf or run gdb, or whathever, but first if something like a contract has failed it will be easyer.
Stepping through code is more like having your nose on the ground.
Both have their merits.
My holier-than-thou take on this topic is: Whenever possible, debug by adding assert statements and test cases.
The only time it is not OK is when breakpoint debugging is overall faster but you are avoiding the hassle of setting up the debugger.
Also OK: adding a console.log or print in your node modules or python package cache.
And btw splunk, datadog etc. is just printf at scale.
Been waiting for "something else" ~30 years & counting.
That counts for a lot to me. Anti-printf people have to go a long way to convince me that it's somehow not ok. I use debuggers every day. I use some form of printf debugging every day. I use whatever affordance I can get my hands on, and I ignore the peanut gallery.
Me too!
I know I'm stirring up shit here but there really are benefits to touch typing (I mean just think about it, using 10 fingers instead of 2 is gonna be so much faster assuming you have 10 dingers)
I'd be more concerned about RSI than speed. You _really_ don't need to type fast for programming. If you do, your tools should be helping you do boilerplate more.
Even that depends on what exactly you mean. I agree with the literal meaning of the words, but for too many people "touch typing" means "home-row touch typing". Which I find extremely awkward and difficult. But literally just typing by touch, I'm self-taught due to StarCraft multiplayer back in the 90s and early 2000s, and have a style that confuses home-row typists when they realize what I'm doing.
So it's more like, you need to type smoothly enough that you're not thinking about typing. As long as your hands are pretty much moving automatically to get your thoughts into the computer, you can use any number of fingers at (almost) any speed, and you're good.
When I got a split layout key it became quite apparent that my technique was "weird" - sometimes I'd notice one hand come wandering over to the other side of the keyboard to find a key it was used to pressing. That became easy to correct once it was so visible!
There are ergonomic issues too. Hunt and peck gives you a lot more flexibility as to how you can use a keyboard.
Most programmers who don't touch type can type pretty fast. There is more than one way to do it, and I'm not convinced touch typing is the right one.
I'd suggest practicing your touch typing a bit more.
But now I'm curious, which thumb do most people use on a keyboard?
I use vim, IDEs, debuggers, printf debugging, whatever works. A tool is a tool. I guess my holier-than-thou position is against the idea that there's one right way to do anything.
Logging is great for long term issue resolution. There's tracepoints/logpoints which let you refine the debugging experience in runtime without accidentally committing prints to the repo.
There are specific types of projects that are very hard to debug (I'm working on one right now), that's a valid exception but it also indicates something that should be fixed in our stack. Print debugging is a hack, do we use it? Sure. Is it OK to hack? Maybe. Should a hack be normalized? Nope.
Print debugging is a crutch that covers up bad logging or problematic debugging infrastructure. If you reach for it too much it probably means you have a problem.
Logging lets you refine the level of printing and is designed to make sense in the long term. There are many technical differences (structured logging, MDC etc.) that I won't get into but they are important.
To me it's mostly about the way you write the logs vs. the way you write a print. A log tries to solve the generic problem so you can deal with it in production if the need arises by dynamically enabling logs. It solves the core issue of a potential problem. A print is a local bandaid. E.g. when print debugging one would write stuff like "f*ck 1 this was reached"... In logs you would write something sensible like "Subsystem X initialized with the following arguments %s". That's a very different concept.
You don't need to pause anything whilst capturing the bug.
Now, if your code is literally connected to running critical systems in an actual factory then you've probably got additional realtime and safety-critical considerations that might push you towards debugging.
But (for more conventional use cases) time travel debuggers can handle multiple communicating systems without causing timeouts, capture bugs in software that interacts directly with hardware devices, etc. And you don't have to keep rebuilding / rerunning once you've reproduced the bug.
Your comment highlights my exact problem with print debugging... You just aren't aware of the tools available to you and you reach to the rusty old broken hammer.