When I got started I debugged using printf() today I debug with print() | Hacker News Reader

When I got started I debugged using printf() today I debug with print()

twitter.com

22 pointstosh58 comments

Yes. 40 years of debugging with print statement (or MsgBox in MFC during that one Windows project in the 90s...). When I'm mentoring juniors, I always see them stepping through the code with a debugger. They're not really understanding the code, they're just looking for the line that crashed ("Oh, there's an NPE here"). And they do a local fix ("We'll just do an if(foo==null) check"). That doesn't usually fix the root cause (Why was foo null? Should it ever have been allowed to be null? Should we catch that earlier in one place?), and they never internalize or understand the codebase.

Printf debugging requires you to reason about the code. And yes, I look like a n00b when I debug something I've never seen before and it takes me longer to find the first bug. But during that whole process I'm learning how the code is really organized and how things go together.

I always advocate for the printf style of debugging.

You can do all of that with a debugger too, likely much faster. Print statements are handy when dealing with an ecosystem with bad debug tooling, disparate systems calling different microservices, etc.

Learning the debugger intimately within your stack is a hugely productive endeavor. Breakpoints are just debugging 101. Learning how to effectively break on conditions, expressions, breaking on multiple threads are vital to programming. Certain kinds of race conditions are nearly impossible to reproduce deterministically with only print/sleep statements.

As an aside, I also encourage a first pass investigation purely with logging/telemetry to determine the issue. This will immediately reveal if your logging is insufficient or not. You can then get down to fixing the bug AND your telemetry so you can catch it without grokking through your code.

Juniors are failing to dig into problems to discover the root cause because they're juniors and that's something that many people have to be taught to do, not because they're using a debugger.

Debuggers are an enormously powerful tool for understanding a codebase and thinking of them as a way to skip over that is a severe misunderstanding of why they're useful.

I always debug with printf for the clear benefits you mention, and I happen to know another one.

Because I am commited to the Way of Printf, I have seen myself anticipating where printfs might be needed, and limiting the complexity of each segment of code to support printfs that aren't there yet.

In my experience, commitment to printf debugging incentivizes coding for simplicity and observability.

Imagine if other engineers had this attitude towards tools: "I don't use a ruler when making a drawing. You have to reach a peace of mind to make a straight line and the junior nowadays just use a ruler instead. "

I say that as someone who programs on paper and uses print debugging a lot.

I have no idea why anyone would voluntarily hammer a nail using a forehead instead of grabbing a hammer - tool made for the purpose.

Imagine that you are in a method X, which is called from 100 other methods. Method X crashes, but you don't know what method the wrong data causing a crash come from. In your scenario you will be putting 100 different printf. While somebody in debugger will just have a look on stack and will see it.

Yep. The debugger is a bloated, boondoggling waste of time except in very special circumstances. It's like a high-tier electronic warfare ship in EVE, when you just need to shoot some rocks 99% of the time.

This seems such a weird logic to me. if you removed the debugger, they’re just going to put print statements wherever until they bartow it down.

Hell if they’re doing that and not putting the breakpoint at exactly where the exception occurs based off the stack trace them already something is odd.

How do you folks debug a third party library without jumping into the debugger? In order to test certain conditions at a breakpoint are you printing and refining the code every time ?

Printf doesn't really require you to reason about the code anymore than stepping through the code does. Usually you already know where it crashes. You can randomly put a printf somewhere but the value may have changed somewhere else. If you are paying attention stepping through the code let's you see when the value changes. As opposed to adding a print, running, adding another print, running, etc. I would also argue stepping through code let's you understand the code better than a simple print. As you can see things happening in realtime. Both methods have their usefulness, both are just tools you should have in your toolbox.

It’s all about feedback loops, and how fast you can make them.

I was thinking about this recently and came to the same conclusion as the tweet. I almost never use debuggers.

Print debugging outputs to a single flat buffer, stdout (or stderr, but the point remains). Where you place those statements matters, because you should be able to predict the output to stdout. If you can’t, you get to update your mental model.

The order matters, the timing matters, the brutal simplicity matters.

Full on debuggers are helpful tools to explore someone else’s program, but mostly I find their only advocates come from those whose feedback loop with print statements is measured in minutes to hours, not seconds.

Of course if it takes 30 minutes to rebuild your app, or 5 minutes to setup the game state your working on, pausing to inspect the state of the world is going to be easier than print statements.

But if it only takes seconds to rebuild, nothing will beat print statements.

There is a difference between debugging my current code - where I can build a mental model and work with print statements and dealing with other peoples (or my long past self's) code. With other peoples code I need to restrict myself to partial understanding of the overall state and thus need a more comprehensive view of the local execution context than what a print statement provides.

Whenever I use print for debugging I end up running the code many, many times. Each time I add another print statement. "Here1", "Here2", "Here3 %s %s %s"...

If I break out the debugger, I run the code once.

I think it's just that people have anxiety/aren't very used to debuggers so print is less stressful even though in reality it takes 10 times more

The very first thing I do when setting up any new dev environment is trying to attach a debugger to my local build

It’s setup time that pays dividends for years afterwards.

I think many folk don’t bother until they need to fix a bug and at that point you are in a sunk cost situation.

Unfortunately, not all languages/systems have an efficient and useful debugger. This becomes especially true when multiple stacks are involved. But the print statement remains possible across almost all tools, technologies and systems

Agree.

Another use case for printf debugging is occasional or transient issues where there is a bug, but it only presents in an aggregate of many runs (or a lot of runtime).

I debug scientific simulations in this way with printf's, because I need to look across 100 log files to find 2 or 3 instances of a bug.

Then I can reproduce it and zoom in on the exact problem with either more logging or a debugger.

Unless you decide you needed a breakpoint in the past.

...and you're not on Linux, because on Linux we have rr! https://rr-project.org/

(I still use print statements 99.99% of the time though)

Use a debugger, you filthy barbarians. Learn how to step through, set breakpoints, watches, and watchpoints. Learn where to use these tools effectively. Print statements cannot give you a fine-grained view into the evolution of your program by stepping through, line by line or even instruction by instruction. Debuggers can.

Debugging by printf() is only extant because the debugging tools generally available under Unix are stone-knives-and-bearskins compared to other environments (Windows comes to mind).

The superpower of a debugger is watchpoints and inspecting data structures. No need to laboriously write out a rendering of the internal state of a program when you can pop open its objects and look at their state in the debugger. Breakpoints, step-in, step-over, step-out, all good things.

That said, a debugger is not a substitute for robust tracing/logging. By the time a program gets mature, it will probably end up being instrumented to generate nice-looking, useful traces. Often, that will be good enough to debug in the future.

I just don't understand people who will actively avoid using a tool. Like, I don't get it. Do you trim your lawn with scissors?

The code that we currently write is at a sufficiently high level that we don't need to reach for a debugger.

I have a local kubernetes cluster with 10 services running, I usually get to the root of the problem a lot faster with a few log lines than having to figure out how to configure my IDE, run the app in debug mode and get all the debug port mappings right and have the right source mappings configured. If the problem is less around business logic and more around algorithms, that's when I spend time configuring everything and stepping through the debugger.

This was one of the more interesting parts of John Carmack's interview on Fridman's podcast:

https://www.youtube.com/watch?v=tzr7hRXcwkw

I thought about this before, as a debugger user. And the trick with printf and think approach is not the printfs but the thinking.

Debugging can be done mindlessly while thinking through what might be going on helps you build up a mental model of execution.

Here's the thing I wrote:

https://earthly.dev/blog/printf-debugging/

One of the things I'll probably never understand is people's discomfort with and/or refusal to familiarize themselves with the use of a debugger and stepping through code. People will spend hours saturating their code with logging, rebuilding and running the code, and then spend more hours pouring through the logs as if walking around with a divining rod trying to guess what's actually happening, when they could simply set a few breakpoints, examine the variables and control flow as they're happening. The biggest "superpower" to my programming skills as a junior engineer was learning how to use the debugger well.

Yes, there are bugs that don't reproduce in your own environment, and yes, logs are a good starting point for those bugs. But, as a general tool to reach for? Not a big supporter of print() debugging.

Logging is just easier. The lazy route is the most common

Because logging is persistent. Once I have good logging implemented for a section of code I can always enable it and see what's happening every time the application runs.

Debuggers are one-offs. Running a debugger (and rerunning the application) every time I'm suspicious about something, or need confirmation, is more effort.

Good logging is having an always-on debugger that's enabled for everybody at once. It's an investment that pays off.

Yes, I know how to use a debugger, but rarely has it been more valuable than good logging.

One problem I have is that I primarily work on embedded distributed systems (multiple software components talking over a high speed messaging layer, using both sockets and shared memory).

As soon as you pause the code in a debugger you "disturb the equilibrium" in such a way that debugging the original problem becomes impossible because remote components start timing out or closing sockets, etc.

I still end up debugging via print statement a lot so I can run the code at full speed. Does anyone have a good tooling suggestion for a situation like this?

My dream is a tool that would allow me to set a breakpoint which would start capturing program state at each line while still running at full speed, but then allow me to step through the recording later. Does something like this exist?

Yes, it's called rr. https://rr-project.org/

This is very cool. Thanks for sharing it.

GDB can kind of do this, you can have commands fire when a breakpoint is hit and continue automatically, and if you want to take it further you can use Python and execute Python code to inspect the state of the CPU and memory in some programmatic way. Not sure if it applies to your target platform, but if it’s ARM it should work.

I do both.

Almost all the inflection points you would want to print to debug, it should really be a debug log level statement that stays in the code. If you're writing a lot of these you might want to simplify the code.

Otherwise yes use a debugger, but question why you're having to debug in the first place. Improve the code quality while you're at it instead of just fixing the bugs.

This is probably going to sound weird, but debugging feels a lot like washing dishes to me. Do more with less and avoid toil.

For me, the most annoying part about a debugger is that it always shows a snapshot at a single point in time, and the snapshot can only march forward. A good fraction of bugs in my code are ones that surface gradually or occasionally over a long sequence of operations. Stopping at the ultimate failure point is far too late, and breaking at every single iteration is far too tedious. But with a log of all the iterations, I can trace the error back to the initial point where the program no longer acts as expected.

That is, I very often find a narrow view of the program state over the whole computation to be more informative than a wide view of the program state at any particular moment in time.

In some cases, GDB allows reverse execution of code:

https://sourceware.org/gdb/current/onlinedocs/gdb.html/Rever...

https://rr-project.org

This is indeed a problem people have with debuggers, so some very smart people found a way to fix it.

rr is so good. "Where was this variable written? Where did this value come from?" Set watchpoint, reverse-continue, and bam you arrive at where the write happened.

Indeed, that fixes the part about it only marching forward. But still, it wouldn't be sufficient in cases where the ultimate cause is near neither the start nor the obvious error, but (e.g.) in the 50th of 100 entries. At least, not unless the initial error is simple enough to be captured by a conditional breakpoint. But even then, I'd probably be reaching for my prints just to determine what a good condition might be.

We'd be hosting the RR inventor to a tech talk soon! I'll post a link here when we have it scheduled

> Indeed, that fixes the part about it only marching forward. But still, it wouldn't be sufficient in cases where the ultimate cause is near neither the start nor the obvious error, but (e.g.) in the 50th of 100 entries.

Not true. rr (and time travel debugging in general) really excels at exactly that problem. You just put a watchpoint on the offending data at the 'obvious error', and reverse-continue. This usually takes you straight to the root-cause. It's incredibly powerful.

(Disclaimer, I'm not exactly unbiased as I work on another time-travel debugger, https://undo.io)

This sentiment comes up often and drives me nuts. Most of the time, the only thing to understand about somebody else's mangled API or shitty vendor-provided library is that their manager crawled up their ass over a deadline so they kicked the can down the road to you. There's nothing else to "get". There are entire languages (C comes to mind) so laden with footguns that trying to "correct your mental model" to comply with it is like lobotmizing yourself so you can deeply understand the mental state of a lobotomite.

There are languages and tools and methods for programming that are effective enough at limiting or eliminating bugs that you don't need much more than a print statement here or there. Most of us don't or can't use them.

Additionally printf statements have direct impact on application itself and can "fix" for example race condition bugs. So on debug it works. On production it crashes. Go figure.

I debug with print statements, and with a debugger and break points. Why make it mutually exclusive? They're complementary.

I debug using printf.

The supposed advantage of using a debugger is rapidly examining the state of the program.

But the debugger offers no means of actually examining anything other than the simplest variables.

With printf, I can create a comprehensive presentation of that state that allows me to understand what the program is doing.

A debugger is great when there's a diffuclty understanding the control flow (Where is the program crashing? How do we get there?).

But as far as examining the state in a way one can easily parse - the debugger won't do that for you. With printf, you create an annotated story of the execution, showing you both the data and what it means.

Example:

    Polygon count: 5
    Polygon orientations: {A: pos, B: pos, C: pos, D: neg, E: pos}
    Centroids: {A: [0, 0], B: [20, 3], C: [7, -1], D: [5, 1], E: [2, 3]}
    Box bounds: {X: [-10, 10], Y: [-2, 4]}
    Polygons with centroid in box: {A, C, E}

Looking at this output tells you what's wrong with the function that filters polygons with a centroid-in-box query (it doesn't handle negatively oriented polygons correctly).

Sure, you can set up your watch variables to present information in that way... but then you're just doing printf with extra steps.

I've found a debugger particularly useful when trying to understand details about other people's code, when changing the source code is not necessarily an option. They can quickly give me answer to how the call stack looks like when a function is called, and let me inspect variables in different frames of the stack.

Sometimes, building / running / doing the flow that produces a bug takes a lot of time. Or the bug is in external lib which would make it hard / cumbersome to instrument with logs. In that case, properly placed debugger will save a lot of time.

If things are easy to isolate, print is faster.

For the next level printing though, do not use

    print('here1')
    print('here2')

But instead write meaningful logs, say

    print('copying X')
    print('zipping to X.zip')
    print('uploading to Y')

which can stay after you're done debugging. (Ideally you should not use print directly but via an util which silences the logs depending on program params, or removes them at build time etc.)

I'm mind blown how big % of code I work with has zero logs and zero comments. Whenever something crashes and the author is not around, good luck.

I usually have a macro called __RIGHT_HERE__ that prints a large arrow, the file name, and the line number.

So why did the author stop using printf() in favour of print()? Or is the missing 'f' a typo?

I think it's moving from C(++?) to python. (Author is the creator of keras)

I personally interpreted it as C years, thus the usage of printf statements versus Python years that have print statements.

My interpretation could be wrong though...

Because c++ can now do std::print() :)

This seems to dependent whether you learned programming In Unix or Dos/Windows.

On Windows QBasic/Turbo C/ Turbo Pascal/Visual Basic/ Visual C++ all came out of the box with a nice integrated debugger.

I use xlog().

My xlog() has the form xlog(<filter>,<format string>, <arguments ...>)

Instead of printing every time, the filter only allows printing in certain circumstances, 16 different sets of circumstances. The format string and argument list, are just the same as the printf() format string and argument list.

The other difference is that the output of xlog() goes to a log-file, not the screen. This allows long and careful inspection of the output, as well as allowing use of grep and other utilities to aid in the quest for bugs.

Think of it as a super-duper printf(). <grin>

I do that exact same thing but the filter is just verbosity level. I end up using my printf-wrapper-with-verbosity-filter for essentially all output, and it serves as a global verbosity knob that goes down as well as up. Default verbose level is 1, and setting to 0 makes the app silent.

Small problem though is a lot of times a nice informative message is constructed of stuff that takes work to do, and the work has to be done and then handed to my dbg(v,fmt,args...), only to be thrown away in the first line of dbg() if(verbose<v) So I have chunks of code behind if(verbose>3) and then dbg(3,...) inside there which just bugs me aesthetically :)

I don't want to think about trying to make something that somehow takes an argument for what work to do instead of a simple value the result of the work. Function pointer? It would be less ergonomic than just if(verbose>4)

Debuggers are nice for development but we also need to be able to analyse field issues. On-disk log files are usually too expensive so we mostly write to raw binary in-memory trace tables, which will wrap, but hopefully will have enough history to figure out what went wrong, and will be included in a core dump.

For issues in the field you really want metrics and logs. That way it's easy to monitor for the state of things and to zoom in on the specific data you need when you're investigating. OMG right now! Or days or weeks from now. With a single entity or local group or a distributed set of them. Even if you're investigating a single system, you may want to correlate with other events in other systems leading to, simultaneous with, or soon following your incident. When people talk about o11y (observability) they mean this.

Ideally, events will be recoverable, but also still debug-able. Depending on the kind of thing you're looking at you may not have the (somewhat dubious) luxury of a core dump.

I'm still on the fence about whether a core dump or a Java exception unwind is more useful for new staff awakened up by a "pager" at 4 am. /s

Have you not heard of Time Travel debugging yet? There are a lot of softwares for each domain - frontend, backend, cloud etc..

One of them is Undo (linux - C/C++/Java/Go/Python), one is rr(Linux - C/C++), one of replay.io(frontend)

The trick is to call printf with no arguments so it will print out the stack frame.

I go on and off using a debugger.

The main issue with debuggers versus logging is debuggers usually help you understand a point in time & space. The debugger just isn't a particularly great tool for understanding what's happened even if you do have a stack traces, and in a modern async world stack traces are often quite broken.

What I do love is tracing, which is an incredible view over time of all the various little subprocesses going on. Attaching some trace attributes to tell me key pieces of data or collection sizes leaves useful clues that future views can see.

It's crazy what my company pays for logs. Our effective price per gigabyte is colossal. And most of our services run pretty quiet, but still it's a huge cost. Meanwhile we must be generating vastly more trace data, orders of magnitudes more, trying us every handler and detail of what's happening, everywhere, and theres no pressure to reduce our tracing.

Tracing gives an enormously beautiful view over time of what's happening. At the same time, logging can still be better because trace views don't make at a glance reading of trace attributes possible. It's still. Better to log because you can select & pick the data you want and view all of it on screen in sequence over time, as it happens. Where-as with tracing you have a very compelling & powerful structured view of execution, but you have to keep clicking in to each span to dig out the specifics. Otherwise I feel like tracing would be a total lock.

I have yet to see another developer who has done local development with the assistance of tracing. I love it, and swear by it, and it's largely outmoded logging for understanding my systems. I usually have such a good view of my systems that I don't need a debugger!

In general it feels like a lot of these tools haven't moved much. Another commenter talked about logging over debugging, because it's persistent & can be returned to, where-as debugging is an ephemeral activity & we don't have tools to save/restore debugging analysis; we re-debug aknew each time we run into an issue. Logging lets us build up a consistent base (albeit at the risk of allowing ephemeral logging to cruft up our environment over time). Tracing seems like a much more powerful paradigm than logging, with much more for free & done consistently, but we still have a fairly unrefined ephemeral system for doing trace-based debugging.

Getting beyond debugging as an ephemeral technique seems like a big leap we need, if folks are really going to find lasting value in their tools.

Ah, yes, good old printf(), the best debugger ever invented!

Reply on news.ycombinator.com