Debug Programs by Modifying Them | Hacker News Reader

Debug Programs by Modifying Them

merveilles.town

52 pointsakkartik36 comments

Honestly, I don't get this debate.

If you were to take the pain points of debug by print, and address them, you'd come up with a debugger. Being able to put those "print" statements anywhere without recompiling, being able to go step-by-step in the code, etc. And sure, that's not always available.

Both end up having impacts on the thing they observe. A print() call can cause microscopic delays that are enough to cause a race condition not to trigger, as can waiting on a breakpoint, etc. If you compile with debug symbols, that can cause some bugs not to surface, as they are only visible when you compile with -O3 or whatnot (eg. a bug involving reading unallocated memory).

Both are just tools. Sometimes one is better in a given situation. Sometimes the other is. I just don't get how this is like an emacs vs vim thing.

For embedded work, I've found great value in being able to observe and manipulate running systems without interrupting code execution. A non-blocking console buffer is very handy, but even more so is instrumentation for periodically sampling arbitrary variables cooperatively / in synch with program execution.

When you're making a 50KW motor spin under PID control, a debugger breakpoint can cause a lot of real damage.

Stepping through can be a pain sometimes, because to find certain bugs it's easier to look through a log throughout a longer run of the program, and see how the state changes.

Of course, a debugger that could generate a log like that instead of pausing execution would give you that benefit too.

Yes, this seems like an odd missing feature that would give me the best of both worlds: a "watchpoint", but that outputs to a log. It may be possible in VS debug or gdb but it's not really .. surfaced?

This is possible in all JetBrains IDEs, just right-click the breakpoint and configure it that way.

In gdb you can put commands to be executed on a breakpoint; you can put necessary prints + continue statement there.

> Both end up having impacts on the thing they observe. A print() call can cause microscopic delays that are enough to cause a race condition not to trigger, as can waiting on a breakpoint, etc.

I seldom encounter a program that is sensitive to a print() but robust to a debugger. Prints perturb programs _less_ than debuggers do.

> If you were to take the pain points of debug by print, and address them, you'd come up with a debugger.

Quoting myself from OP: The key skill it teaches is reasoning about a program by modifying it.. When might better debugging tools be counter-productive? If they cause you to forget that there is a debug cycle, and to stay too long observing when a new modification would get you to the answer faster.. Tools will always will have limits.. Prints prepare one for times when you have to leave your tools behind and enter the wilderness.

Am I understanding this correctly? Are you saying it's better to change things rather than understanding the existing code?

I had a coworker once who was proud that he didn't know how to use a debugger. It was something about being able to statically analyze a program without dropping into a debugger that conveyed his superior skill. He only used print statements.

For me personally, I use both print statements and the debugger extensively. When I'm wrangling a hard to understand bug, I'll litter my code with print statements and then once I find a problem area I'll set a breakpoint. When I have more information, I'll even set a conditional breakpoint - I can't imagine wading through a lot of print statements after a few iterations when you know exactly which conditions will trigger your bug.

The tweet is not advocating print debugging per se, but about understanding the code by modifying it. This is how I approach any foreign/old code all the time, I can recommend it. If you have a theory about how it works, edit it, and see if your edit works expected. It may or may not, either way you learn something. I love debuggers, but editing the code can also help you trigger a breakpoint just before that special case hits. Much faster than conditional breakpoints.

I am always surprised by how a significant part of the population are purist or religious mindset about everything, specially tools. They take a rigid hard stand,idealized to Totem status and don't move from that as they consider anything else "Tabu".

When I read "debug by print VS debuggers", I wonder: why versus? Why you can't use print AND logs AND debuggers AND REPLs AND asserts AND tests AND analyzers or whatever other tool whenever you need it, depending on the problem you face?

We use all those tools and way more, and it is probably the flexibility and ability to use all of those what gives us an advantage.

It is probably the effort to learn something new until it makes sense, most people just don't do the effort. I remembered the first time I saw asserts everywhere in one of the best programmers' I have worked with code. It looked ridiculous from my old mindset but I said to myself: This guy is really good so odds are it is me the one who does not get it.

So I started just doing the same the guy did, removing rational judgment for a while, so I did not let my brain to trick me into not doing the work finding this or that reason.

After a while, it made all the sense, my code become much better, I had almost no bugs compared to my early life, and bugs were catched instantly near the place they were and did not produce secondary bugs that were the ones that I catched before, taking me a long time to decipher.

Remember that reason alone is very poor for understanding the world. The world is a complex place and reasons are multiple and nuanced. You learn much better if you just copy what the best people do in the field and experience it yourself like kids do with their parents, then you will understand.

You will not understand by arguments alone. Your arguments could be brilliant, but you are always modeling the world, and if your model is wrong or lack essential constituents, it is garbage in, garbage out.

> Remember that reason alone is very poor for understanding the world. You will not understand by arguments alone. Your arguments could be brilliant, but you are always modeling the world

This is heresy in geek circles, where people believe they can reason from first principles and background knowledge through entire other fields of study without ever having to discover contradicting facts. But it's very true.

Sometimes you can't even use print debugging.

I've debugged by modifying colors a few times.

With shaders this is often the best bet. You end up encoding some value you want to measure as a color, and you can observe how it varies over what you’re drawing. There are some pretty sophisticated step debugging tools these days, but just like with print debugging, debugging by color always works on all platforms and has no special setup.

Multiple times I've had bugs that disappeared when print debugging was added. To say that that was frustrating is an understatement.

(But where there's a will, there's a way. It just takes longer, sigh.)

Those are known as Heisenbugs.

https://en.wikipedia.org/wiki/Heisenbug

Sounds like you solved the problem though. Leave the printing in there:P

In multi-threaded code or using interrupts (ie. clock or I/O), it's pretty obvious what causes this. Fix can be hard though, but absolutely required for robust execution.

In single-thread code, I've often noticed my assumptions about the code paths were wrong. So print debugging actually improved my understanding of what the logic really is doing. Then, either improving the process, or my assumptions/memory of it. Many times debugging like that removes subtle bugs and mistakes that otherwise would've remained invisible and dormant.

Usually when something is off about the execution, however long it takes, it's usually my own assumptions that's wrong. The program logic just does what it's told, and is usually void of off by one errors and the like. If there's subtle bugs, it's often shallow object clones or mistakes handling complex datastructures that seemingly works.

Debugging is just a tool though, use whatever suits you best. I do like the idea of improved tools like RR. However, you often delve into languages that don't have them, so print debugging remains a useful hammer lying around.

Fully agree with this. More often than not bug is in the computer between my ears rather than the computer on my bench.

One trick to minimize debugging footprint is to store logs in a ring buffer/array list and defer printing. It's not perfect, but if you use a ring buffer, you can avoid making any syscalls until you are truly ready to print. Thread synchronization can obviously still be a problem, but that can be mitigated by using thread local buffers or by using a lockless ring buffer.

Ring buffers are also nice because you can also use them for cases where you don't normally want some information printed because it would introduce too much overhead, but you want the option to print it out in case of crashes/errors. On an engineering side, you just need to be careful not to put too much through this system so the buffer stays useful and also avoid putting something only in the ring buffer system that should really just be logged normally.

This is genius. I hadn’t heard of this logging style before. I’m going to try it tomorrow morning.

A variant I developed in an embedded system, which had weirdly defective debugging facilities: define an area of memory which is not cleared on startup. Log into a circular buffer there. On startup, output the contents of the buffer over the serial link (or similar).

There are many variants. One is to put the ringbuffer in a shared memory segment or mmapped file, and use another process to read and print the content.

You had colors? Back in the day, all I had was an unused GPIO pin and an oscope. One pulse, assert code 1, two pulses, assert code 2, etc.

(I’m kind of riffing on an old Dilbert cartoon, but debugging with a scope was common in my life in embedded systems.)

Are you sure it's a Dilbert cartoon and not an XKCD? https://xkcd.com/378/

I enjoyed the XKCD. Thanks for that. I'm currently interested in Cynefin and complexity theory in organizations, so it's kind of right up my alley.

Here's the Dilbert I was thinking of:

https://dilbert.com/strip/1992-09-08

Once upon a time I had an attiny85 chip with no serial interface on the board. Lacking a digital logic recorder in my home lab, I debugged it by blinking a diode. Time consuming, but apparently has worked :)

In my teens, after long programming session, my brain simply refused to understand what I was looking at. These occasions often led me to try to fix bugs by randomly changing the code. Some bugs were mere off by one errors, then I changed for loops to stop earlier, initialized pointers to NULL and early bail out of a function if a parameter was NULL, increased size of arrays...

It worked more often than I'd like to admit. Reading the code later was utter incomprehensible and left a bitter taste. Eventually I stopped using such technique. It was better to leave to code crashing than having it working without knowing why.

To this day I still see inexperienced C programmers making the same mistake. Such fixes are the kind of mess that will make something work on a computer just to crash in another OS or architecture.

The funniest example of such a specimen I saw was a very interesting graph problem solved in C. It worked beautifully. The algorithm was elegant and low complexity but it was not general enough and some cases had to be specially handled. New special cases were discovered and the team had to resort to a few tests that started breaking when other cases were fixed. Bug appeared and were fixed randomly. A last case was fixed and then all the other previous important cases crashed.

What to do? Valgrind to the rescue!

After running the code on Valgrind, success! Not that it helped fix the bugs, there were lots if complaints, but it didn't crash. So, instead of fixing the bugs, the team decided that it was to be run only under Valgrind and everybody was happy.

Oh hey man, surprised to see a Merveilles link on HN haha :)

Debug by print/console.log is fine -- sometimes it's the fastest/easiest way to quickly troubleshoot a fairly complex flow, etc.

I've worked on mobile web apps where we had no debugging tools (back in the early days) perhaps due to lack of full access to the hardware, etc. So we would do things like set a background color based on a certain value, add borders, change font color, stuff like that. Or just draw debug console text on top of everything.

Sometimes the actual debugger for the provided software/platform is of low quality too.

Conversely, being able to set conditional breakpoints is pretty powerful and I miss doing so when it's not possible!

Here's the thing: Use the best tool for the job. Work smarter, not harder. I'm debugging something right now. I can use the rapid compile time of golang to do things like: change the binary format, so that I can trace what part of the buffer was written by which function invocation. I can also write very sophisticated conditional debugging that would be iffy on most debuggers, where the machine sifts through the possibilities for me. That "one weird trick" has earned me a reputation as a magic debugger multiple times.

Sometimes, it's best to use a printf and spot the thing in the console, or to see what's going on with the data to gain a better understanding.

Sometimes it's best to debug something to rapidly see a complex interaction firsthand.

It's always good to be smart about using multiple tools creatively.

Print debugging is okay, if you don't remove the print statements when you are finished. This technique is called logging or tracing, and is nothing new.

It makes total sense to insert a lot of logging statements into your code, which can be switched on when you need to diagnose something. This can be also used then in production, where you can't just change code (but switch the log level in the config). Very useful

Adding and removing print statements (== modifying code) as you are investigating is bad practice, because you are modifying the program you try to fix. It can be a lot of clutter inside the code, and maybe you fix/break the program by coincidence and didn't even notice. Once you undo your changes, before committing the bug is back.

honestly, when I approach programming, a primary concern is to spend effort to help make the program easier to modify. I'm not the original proponent of that as a goal, its just something I read somewhere but it really helped me in my career.

I have different styles of debugging dependent of context.

If I inherit code or libraries from someone else, I use the debugger without shame.

For my own code, using the debugger is a sign something is wrong. This demonstrates a need for extra logging: How can I troubleshoot software in production in the field if I can't do it on my own machine.

Print style debugging is usually reserved for interfaces between my and someone else's code, to clarify the contract.

None of this is a religion of course.

Ah, a merveilles post! Haven't seen one in the wild before, only heard of it through the Hundred Rabbits site, great to see it.

Seconded. Anyone got a Merveilles invite?

for poorly written python code that is just too large to add the many print statments.. I run it using trace and pipe to a log.. ofc pdb is awesome too.. provided it crashes so I can work in the stack trace..

for bash I do something similar by using set -x and a custom PS4 prompt that gives me file function and line number..

in my own code I have info, warn, error and debug log lines (for both bash and python).. so I rarely have to resort to enabling trace..

Reply on news.ycombinator.com