“I think the vast majority of developers still debug using print() statements”
twitter.com
twitter.com
Rarely I run gdb when I'm debugging a particularly hairy C issue, but with any other language I don't even bother and add prints everywhere. The JS "debugger" statement is pretty useful, but it's still very awkward compared to the golden standard of Microsoft IDEs from two decades ago.
Don't blame the users, blame the tooling, editor and language developers that never cared for delivering a good debugging experience.
Nothing on other platforms combines the power and ease of use.
That's why I quit using it and went back to debugging with print.
Print-debugging limits my cleverness and forces me to think about what's going on.
The only problem is my current projects are all on Linux.
So I figured "When in Rome" and I have Visual Studio and I expected I would come away saying this is pretty cool but I want a modal editor.
Nope, it's terrible. It does have a debugger that is better than I'm used to, but I spend very little time in a debugger, what we're doing isn't rocket surgery. Visual Studio apparently doesn't understand how git works or doesn't understand how files work, so e.g. I'll modify somefile.cs in one tab to fix something, compile, verify my fix worked, go to the Git tab and Visual Studio is wide-eyed. What could I want? There aren't any changes. Yes there are VS, you just saved them, to a file.
So I exit VS and restart it. "Oh!" realises Visual Studio, somebody changed the file. Yes, it was you. You wrote to the file, when I hit save. That's what files are Visual Studio, why is this hard? Now I can commit my change.
C# is a much better language than I expected, PowerShell is a much worse language than I expected. Lots of surprises, good and bad, but I think "Visual Studio is bad at the thing it's for" was the largest surprise.
It sounds like using a second program to add and commit files to git would work better for you.
(I've not spent much time with visual studio)
Nor brain science either, I'd hazard.
https://www.cambridge.org/core/journals/behavioral-and-brain...
Although that's mostly because BBS was ahead of its time rather than because of the topic specifically. BBS looks a lot more like how you'd design such a thing if you started out in an era with the Web, rather than dusty printed journals, even though BBS actually is a printed journal and started in 1978.
Anyway, Stevan Harnad actually taught a class I took (as a postgrad, so not pass-or-fail, I was just showing up because it was interesting). Cognitive Science is definitely Brain Science, but it's true that I am not employed to actually do that.
It used to suck not unlike you describe and there is a slight hidden gotcha - you need to select git as your source code provider in the settings as VS supports other version control systems as well.
For MSVC, I had to write some visualization files (.natvis), phpStorm displays everything just fine.
Also since were on the topic I started using pdb++ and my debugging got incredibly more efficient.
And yes you can use it on frontend stuff if you invoke your browser correctly
And yes you can use it to debug mobile through chrome's inspect feature. And yes you can proxy that to use it in your editor of choice and yes GUD with emacs is supported
And yes the "debugger" keyword correctly triggers the breakpoint through this entire pipeline. It's borderline magic
The concept of remote debugging is just one of instrumentation, that's from the 1960s, this isn't a conceptually new thing. It's a functionally useful one
ndb index.js
But node debugger died at some macos release and I haven’t figured anything close to it.Thing that's nice with ndb it's the same UI as Chrome devtools that I'm so used to.
Debugging via Webstorm felt too weird (tho probably 2x more powerful). Not a VSCode user.
- you set it up correctly
- your entire stack is compatible from end 2 end
- there is now weird thing with map files
- you don't dev using firefox
It's a lot of if.
So sadly, lots of print(f"")'s
I know it's a problem with Python and Node though.
Using print is a sign tooling isn't up for the task. Not necessarily inexperience.
This. For application developers, if your users aren't taking advantage of features in your software, it's a sign the features either aren't actually better, or aren't intuitive enough to gain traction. It's a tell that you still have work to do.
I don't see why the same arguments wouldn't also apply to tools for developers like debuggers. Just because your userbase is more technically savvy doesn't mean you shouldn't care about ease of use.
In my opinion printing or logging is much more useful than debugging when you want to be able to understand the sequence of events leading up to a bug, where the debugger isn't very strong. After analyzing the logs, using conditional breakpoints depending on each other can help you catch the issue live.
Each tool has their strength and weakness, but I would never go so far as to say that debuggers are pointless. Being able to inspect memory and browse for inconsistencies can be priceless.
Do these support edit and continue?
Last I played with XCode, it didn't.
Major flaw.
Both edit variable and hot reload code.
While at it, you can also change the value of any visible variable and evaluate any expression in the context of the currently executing scope.
In fact, I often use pdb anyway because it is sometime just quicker to run, and the code execute faster under it. Also, being able to execute any python code in a shell like env is super nice, and the concept of having commands instead of looking for a button has the same pros/cons than the cli.
I wish there was a 'here be dragons' statement that you could put and it would run a function or a snippet on a special debugging mode (without the need for a debugging tool). And then ask questions like "who set this to value X" or "just print the local vars here" etc
I think the closest we have to this are the debugging environments that (for example) Flask gives you (maybe other frameworks)
But there's no operation tracing. I'm not aware of any language/system that automatically logs all operations with their context/value and boils that down to a useful display.
Debuggers seem to all use the ancient machine code model where they replace an instruction with "And now launch the debugger to see what state the variables, memory, and registers are."
It would be useful to have more print-like features that create a searchable list of all entries/exits/creators/destructors in sequence with associated parameters and values.
Do blame the users. If they would've said "nah, I'm not using your programming language, it doesn't have a decent IDE and debugger", life would be much better.
I'm not sure where the ecosystem went wrong, but it's strangely related to the web.
What kind of a strange beast was that? :)
I do remember fondly debugging in Delphi, but when using Perl and Python's debuggers I don't really miss it
I do, however, know plenty of brilliant developers who can make interactive debuggers sing and respect different people preferring different approaches.
Note: If any of my perl code upsets the debugger it's because I didn't realise as a result of my preferences and I'll still consider it a bug.
Half a gigabyte of ram was, and still is, a gargantuan amount of memory.
That doesn't sound right. The "100 MHz" in the first half is too low for a machine from 20 years ago, and "512MB of RAM" for a modest machine of the same era is too high.
Point still stands, you don't need 8 GBs.
Insanely fast and capable debugger for Windows. https://www.youtube.com/watch?v=r9eQth4Q5jg
(I have no relation to the project, just think it's great)
1) Well, okay, there are exceptions. Perhaps you have to find a spare IO pin attached to an LED, and make it blink at you in morse code, because your serial port hasn't come up yet, so you can't print to the console; but it's the same idea.
Then you learn debugger like once? twice? thrice? And still they are not this complex
>but one has to balance the time spent learning & managing fancier tools versus the time spent getting the job done.
There is funny saying which goes like this
You can save 2h of reading debugger docs with 12 years of using prints
Personally, tcpdump is my favorite debug hammer, but printf is second. FreeBSD's kernel debugger is pretty neat (especially over an IPMI serial console), but fills me with dread when I need it.
Maybe just a spare IO pin and an oscilloscope.
I made a write-only UART on an ST7. Clocking worked out to 500kbit because that was the first configuration I found where I could make the mark and space bits take an equals number of cycles. I still remember thinking: why in the world is the timing for each of these instructions so different?
Seeing the state is faster with a debugger is probably faster, but the process is an analog to explaining the situation to a rubber duck.
Logging, done well, will inform (1) exactly what version of your code you were running 3 months and 12 days ago, and (2) where the potential issue may be.
Therefore it will inform you _where to place a breakpoint_ should you wish to repro the issue and debug locally.
Good luck getting started in a local debug session without having had adequate logging in place.
With the debugger, you can only see the application state at one specific point at the time.
With added print statements, you get a nice history of the application state.
If I already know where exactly something is going wrong: sure, I'll could a breakpoint there, if needed - often it's not.
If I don't know where exactly the problem is, I'll add print statements.
Really depends on the debugger. Some tools support “time travel” debugging, which essentially stores a structured log of memory state over time.
(That wasn't what I wanted to post; I should have waited.)
More like stop the flow of time, fix whatever went wrong, fix why it went wrong, go back before it happened, and then restore time flow, as if nothing had happened (so more restart before than resume after).
I don't know if that's what the OP wanted or not, I'll leave you to explain.
Just yesterday I debugged a problem starting with gdb and ending with print statements. I needed both to find the issue. This isn’t an emacs vs vi debate, I completely agree.
Logging + threadid + timestamps and something to organize/sort the output is about all you can do.
Debugging parallel/multithread code is a badly unsolved problem in programming, and something that TDD, which tends to focus on single thread units, also doesn't really address.
I think it would require something that can built on-the-fly visualized reconstructions of what happens, almost like a starcraft game where each unit is a process/thread and it replays events from the log in reversible forward/backward slo-mo. Interactions or message passing can be attacks on other "units".
Or some dynamic graph/process diagram builder that links together the processes when message passing occurs or flashes/changes color as interal process state changes.
Good news everybody! All your code should be built with the intent of multithreading or parallelism to some degree, since that is all that we can really do with the remaining scaling from Moore's law, and really has been for a decade now.
Edit: oh I forgot. Debugging multithread isn't even a single-VM issue now. Even more common is visualizing interactions across many many VMs thanks to microservices.
There is a partial order that is fine and there is a partial order that is not fine. If we could define the orders with a sorting function and apply them to events in real time that would be perfect.
In effect my partial order cannot overlap your partial order. This is the grounds of the YATA algorithm.
Happens before relation is really powerful. But it needs to be visible to other processes.
But I see how once you have to cross the rubicon to parallel processing, to make it sane, you'll need to rethink the old ways from simpler serial execution programming.
Every thread monitors every other thread's thread Id and the lowest thread Id wins. When a thread finishes its work it updates the thread Id to Integer.MAX_INTEGER so someone else can have a go.
Readers block writers and writer's block readers so it's not very good. But it's a start.
It's hard to update a linked list as you could accidentally partially update it while it is being iterated. I could allow parallel edits and reads if the read is after the edit. So I would need to introduce a different variable for checking while reading. It needs to be logically after the write.
HTTPS://GitHub.com/samsquire/compare-and-swap-sort
Nobody is saying that you should never use printf debugging. Sometimes it is the best way.
And debuggers have their uses too.
Printing is my usual way of debugging. I use / switch between various languages that have unequal support for interactive debuggers with different interfaces, in various contexts where debuggers are not always usable. Printing / logging always works the same regardless of the language and the context.
Step by step often looses me, and print forces to think about what and where to print so it also has its advantages.
Interactive debugging has a big advantage: no need to recompile / reload each time you need to check out the value of a variable at a new location in the code.
But yes, the fact I switch between languages and contexts seems to encourage me to use generic tools that work okay for every language and context instead of specialized tools, so Kate and printing it is. But you can become efficient anyway, and for a task where printing is not enough, you can always spin up the heavy tools.
this exactly.
Print based debugging is a great way to verify all sorts of assumptions, and it gives a wonderful order based view of execution very quickly.
If that turns out not to be enough - by all means, break out the real debugging tools, but I'd wager 95% of the time, print based debugging will lead you to the answer more quickly and easily than spinning up gdb or inspect-brk or visual-studio or any of the other heavy weight tools.
Not that those tools aren't incredible (I mean that so literally) - it's just that they're like bringing a back-hoe in to plant your tulip bulbs... it's complete overkill for answering questions like "Did this code even run?" or "how many times is this function getting called" or "is function a called before function b" and SO many other useful basic tests for my assumptions.
This is a very good point. I have taught beginners before, and saw many of them would somehow open the debugger in their IDE, then start stepping through all their code absentmindedly, as if the mere act of doing so would somehow cause the bug to become apparent. By the time they see something wrong, they've already gone well past the point at which the first error was. I have called this phenomenon "debugger tunnel-vision". It seems like the debugger is more like a ritual to them, some sort of magic that will automatically find bugs quickly and easily, and they get immensely confused when that strategy obviously doesn't work.
I turn them towards using prints, and many times they find the bug even before the next run of the program, while considering where and what to print.
IMHO the "tool fetish" that some programmers have is actually counterproductive.
Imo you've blamed tool fetish when it's really inexperience that is the problem.
So
> Tools help you be more productive. The keystroke reduction in using debuggers far supercedes print debugging
Sometimes yes, but in general I have seen those tools get into the way. For some reason I rarely need them and juniors often get mislead and don't see the forest for the trees.
I have found that reverse debuggers (rr, pernosco) fix the tunnel vision issue by forcing you to identify something wrong ('no, this shouldn't happen') and then letting you work your way up, constantly asking questions ('ok, where is the next thing that doesn't make sense') instead of passively hitting F8.
Basically, debugging is an active process, which requires a proper strategy, and print, code reading, and reverse debuggers all force you to have/design a strategy. Regular forward debugging doesn't in many cases, especially when you are not familiar with the code or the usual strategies you apply don't work in your slightly devious particular case.
I really dislike these articles that look down on people for doing things their own way when that way isn't actually wrong. There's a lot going on when programming, and there's very rarely a decidedly wrong answer.
Debugging using just debug tools in the IDE is just scratching the surface.
A demo I like to do is Azure Application Insights, which takes memory snapshots of web server processes in production whenever they throw exceptions. Combined with source indexing, it lets you "debug" a release build from a month ago in a matter of seconds.
Just recently I was trying to do the sales pitch to a semi-technical product manager, and he mentioned that they were seeing "occasional login issues". While on a video conference session we managed to repo the issue: after entering a valid password, the application promptly reloaded the login prompt without any error message.
Sit down and think about how long you would estimate a bug like this would take you to fix: "Occurs in prod only, infrequent, no error message, just does nothing."
App Insights helpfully had a "debug snapshot" available for the error. It was literally a single click to load it in Visual Studio. It jumped straight to the line of code that crashed, where there was a comment along the lines of "we may be too lax in parsing this". The snapshot let me inspect the variables and... yes.. a bit too lax they were.
I did this in about two minutes while chatting with the manager. Boom. Solved!
I've had similar troubleshooting sessions in the distant past that took weeks or even months to resolve, because I just didn't have the tools available to me.
Tools make all the difference, and anyone who argues against this just doesn't "get it" in my opinion. I don't care how wizard you are at shovelling, you aren't going to out-perform the back hoe.
(If it’s just a pitch, it’s a great one! I just don’t use Azure currently).
The point is general. Any time you hear terms like "time travel debugging", you should get excited, because someone out there has worked very hard to make your life easy.
It's just ridiculous how much effort has gone into some of these tools, and then has been ignored. It's a waste that bring tears to my eyes. You have all been given light sabers, and you hit each other with blunt sticks.
They're not ignored, most are just not feasible in production due to both performance and memory overhead. They're still used for tests to check for issues (valgrind?)
If you can point to something that actually works properly in production without much overhead I'd use it in a heartbeat.
Also, I'd say proper logs are a form of time traveling debugging... so a system built on logs might work :)
If time-traveling debugging becomes table stakes then we will build systems that can only be debugged by time traveling debuggers. Specifically, because we'll lose our healthy fear of complexity beyond what logging and a normal debugger can manage. And that fear (more often called "experience") is about the only thing that causes the greybeards to push the non-greybeards for simple, elegant solutions.
I think we collectively go further, faster without the magic because magic makes us willing to discount complexity.
But I am something of a Luddite.
[1] By my father, a now retired Social Security Administration manager, roughly 1980 to 2015.
80% of the time you want to know what's 10cm below the surface, and a shovel is every bit as good (better, since you don't have to go and start the backhoe).
If you're in the position of working on simple problems, it's very hard to justify the work of learning these tools. Back when I worked on a multithreaded database engine, I would have killed to have `rr`, but when I'm trying to get simple procedural code right it really does seem like a lot of hassle.
With proper logging, a few days?
- pull up bug report
- figure out user from logs usually based on what he was doing plus IP address + time issue ocurred
- look at what he was doing and what went wrong
- look at code, see what could possibly go wrong, fix all found issues, apply patch, test and deploy. (alternative - see what recent changes were done there from last few deploys to find possible culprits)
Now imagine small issues at scale, millions of users, lots of servers and services, complex architectures... where you can't even reproduce. Those are fixable with logs. Debugger won't really work unless you get a crash dump. Memory snapshot as you say might work - but to get the snapshot I assume it pauses the service to generate memory dump and takes it offline... so equivalent to crash/assert?, otherwise I'd be very curious how to get instant snapshots :)
It’s not perfect, but it’s close enough to be indistinguishable in practice.
It “feels” like you attached the debugger to the production process. You get the heap, the call stack of every thread, etc… It makes it trivial to inspect things like “what was the database query and what result set did it return” or “which specific heap object was null”.
That is a neat idea, I'll definitely use it in the future where possible.
> The original “keeps going” unaffected
That's not really the case, copy-on-write pages means any write after the fork requires a new page in ram.
That will mean increased memory usage and slowdown of writes in addition to fork costs.
On something that uses a lot of memory and writes a lot it will be significant.
If the service is already optimized to run with just barely enough ram to keep costs down then this fork-and-dump is going to oom in prod.
> It “feels” like you attached the debugger to the production process.
Yeah, memory dumps are very helful, gdb + dump usually means bug is fixed barring any corruption that happened long before point of manifestation or business logic issues.
You have to understand what the tool does, how it installs, and if it costs money, lordy you have to wait for it to be approved by 10 levels of management.
Another problem is that log indexers and the aforementioned Azure insights almost certainly would require boatloads of storage and processing costs. Programmer time is fundamentally disrespected in organizations, and debugging/QA is even moreso. Companies basically throw a laptop at both of them and say "good luck with whatever free tools are out there".
But your example was basically a stack trace, from Javaland, I wouldn't call it revolutionary in tooling and is ground-level features of Javaland IDEs for about a decade or more, but maybe C#/whatever is different.
I won't disagree with the core point, since I complained about a lack of visualization tooling for multithreaded apps above. We need more tools and tooling awareness.
In my experience it mostly happens when people that use very basic dev tools (e.g. Vim and GDB) set up a project. They don't use a proper debuggers so they don't take the effort to make it easy to use them, and it's often very difficult to retrofit later.
We did all kinds of things like like custom performance metrics for processes.
Then in my next gig we used sentry.io. Its ok as far as it goes but it's not a quarter of the tool AI is.
Related: Full story is an amazing front end recording and troubleshooting tool.
* I usually fix the bug right there in the debugger where I find it. It's so much easier to fix logic in a REPL with all the actual variables available then in a slow guess-and-rebuild cycle.
* Being able to inspect complex objects is so much more convenient when you can right-arrow your way through the levels or watch a nested subpath.
* I'm constantly surprised and confused as I make my way through code. Reality contradicts my assumptions at every step so I need the ability to see that to find my way to the bug at all.
* I love log statements, but I typically need the debugger to find the place to add the log statement. There are so many candidates I'd have to add five prints for every one I use.
This is a huge point that a lot of people miss. When I was a junior dev working on some shrink wrap software, I’d submit a change for code review, and my stickler boss would read it and then say “have you single stepped through the main path of this code yet?” I found this annoying, but he made me do it, and god damn it if I didn’t often encounter situations where the code apparently worked, but I caught subtle errors by watching the state change one step st a time. “Oh my god, this value is never even looked at and I’m getting lucky it never fills the buffer on subsequent iterations!” It’s very, very eye opening. It became clear that it’s hubris to read a piece of code and assume I can interpret it faithfully. The debugger gives you another picture of reality.
Does it crash? Debugger, backwards from the crash.
Is there some "wtf is going on 'here'" with a pretty undefined 'here', and only sometimes under unidentified conditions, but likely algorithm-related? logs, assertions, printfs. Until some more clear idea of the source of the problem manifests.
Does the former turn out to be truely bad, in the sense of unintended side effects e.g. by something writing beyond valid addresses and it is known what is being overwritten? Debugger.
Finally, is there some misbehavior in some already identified range of code? Debugger.
Of course the environment matters, too. Build time vs. debugger features available.
Most effort/time is needed to solve wtf-style problems. As such, a significant part of the time/effort can plausibly spent for some form of printing.
The last kind of problem (bad part of the code already known) doesn't usually need a debugger: If the faulty part is already narrowed down, it is easy to spot the problem with eyes in most cases (at least knowing the intended behavior, maybe not when looking at some random unknown code).
Summary: More problems can be solved using a debugger quickly, but more time is needed to trace down those problems that cannot.
Plus, saving the information in a file is very handy, because it lets me examine state in some detail, with split editor windows, grep commands, etc. Quite often I'll preface the print() output with special tags that I can use to sift through the log in an editor or, in more complicated cases, with code that analyses the condition numerically or graphically.
I've been doing this for several decades. It ain't fancy, but it works. Compared with debugger actions like breakpoints, this helps me solve my problems much more quickly, and with greater insight.
Reframing Graham's statement at an individual level, print() is what I use, the vast majority of the time.
I don’t want to suggest that tracing or whatever the system you describe isn’t useful, but I think the main advantage people get from a debugger is being able to look at the values of things and step through the program execution (and often find that it does some surprising things).
I use print's for most everything else. The reason is pretty simple - the print's customize the dump for the specific problem I'm trying to track down. Using loggers generated far, far, far too much noise and not enough signal.
When the problem is found and fixed, I use `git diff` to find all the added debug logic, and delete it before checking in.
I use "TODO" with a specific suffix to mark all the places in the code that are of interest for the work at hand (unfinished code, debug logs, places that might need modification, etc.), so that when I want to get a glimpse of what's left to deal with, a simple search for this mark yields all the places, and just them. Then once I think all the code to commit is ready, before doing the final diff review, I do a search for the mark and remove what's not to commit. It's much easier and safer that having to re-discover transient code through review, since you don't have to switch between review and edition for each cleanup, and you don't risk to miss some that would camouflage in a new chunk of code.
Someone who steps through every line of code will probably take longer to find the bug, and will spin cycles on rabbit holes and false paths.
Someone who uses “runtime debugging”, or rather has a hypothesis on the bug or what a piece of code is doing and uses a debugger or print statement to verify that will find the bug much quicker.
With this mental model, you can see how print statements can be as effective as a debugger in many scenarios.
I just don’t see where this skepticism against specialised tools for tracking down bugs comes from.
Probably from the previous skepticism against print debugging.
Every prevalent strong opinion in a social setting is eventually met with an equally strong opposite opinion in the same social setting.
I work with multithreaded code and lock free data structures.
I don't use a debugger. So I rely on having a model in my head that could be wrong and what I plan to do is analyse logs after a test run. These are guaranteed interleavings that actually ran.
When you have 100 of threads and random interleavings your mental model is problem is necessarily different to thinking I'll reach to the debugger to tell me what the program is doing.
When you want serializability you need to design your system differently. I implemented multiversion concurrency control and use timestamps for correctness. I have yet to write a TLA model for it though.
https://github.com/samsquire/multiversion-concurrency-contro...
I also want to test my Raft implementation which isn't properly threaded. But I also need to test pathological cases such as old raft nodes coming back.
Most of the time you're reaching for the debugger is that you don't understand why your web framework got into such a state where something is null such as your Inversion of control Java container as the control flow is so complicated. It doesn't help with multithreaded programming.
If two threads (or computers) can simultaneously be at any point in execution, you need to lay down some baselines for future execution. If the baseline is not met you need to Thread.yield or busy wait or busy wait and then park yourself for scheduling by the OS.
So you need to wait until the system state is valid to modify - there is nobody else modifying state. I implemented the left right concurrency pattern but it causes duplication of data. One of my ideas is to combine it with arrays of structures and structures of arrays to at least benefit from the duplication of storage for parallelism.
Once you got past that check, you do your work and then compare and swap and check that the world is still valid - there might have been someone else who was faster than you.
I'm not an expert I am just interested in multithreading and scalability challenges.
If there is multiple changes, such as a doubly linked list. I think you need multiple CAS and you need to update your guard to guard against that partial update too. How do you avoid a partial update being read? You could offload the check to a reader of the linked list to check if there is a partial update on that node.
These problems are difficult to debug but you need accurate thread interleavings that you can sort into total order which is difficult as some things are going on at identical times.
I've been reading Go scheduler sourcecode go/proc.go trying to understand how it multiplexes G goroutines onto threads M. Lots of locking going on!
I am interested in bloom Lang too for distributed programming.
Still, it remains easier to print a value than to reach for gdb. But once you have gdb going, it is easier to do in depth fine surgery with gdb than littering your code with printfs.
Now it seems the two worlds are entangled and I'm confused.
C-x #n
Indeed. In 1979, the debugger in Version 7 Unix, called "adb", was quite primitive by today's standards. For example, you couldn't even set a breakpoint at a specific line of a C function, only at the function's entry point.
For the "adb" documentation, see Volume 2A of the Unix Version 7 doc: https://plan9.io/7thEdMan/v7vol2a.pdf
On page 314 it says: "C does not generate statement labels. Therefore it is currently not possible to plant breakpoints at locations other than function entry points without a knowledge of the code generated by the C compiler."
I.e., you'd have to delve into the machine code to find the address of the first machine instruction of your C statement if you wanted to put a breakpoint there.
Debuggers have come a long way since then.
For my client (native Apple) work, I use breakpoints all the time, but they are not always that useful; especially when working in a threaded/time-critical environment. I often add print() statements, to display aggregate and calculation results. Swift has a fairly nice reflection system, that can be used to rabbithole a lot of information.
When running in debug mode, my console spews out all kinds of stuff. It can be a firehose, so it works best, coupled with breakpoints.
Been debugging for over 35 years. I’ve learned use whatever tools are available, but my most useful tool is in my noggin.
Those of us "of a certain age," that have done firmware development, may remember ICEs (In-Circuit Emulators). These would probably be impossible to do, these days, but they were about the best debugging tool you could ever hope to have.
That's what optimization does. If you have `x := 1 ; y := 2 ; z := x + y ; print z ;` optimizing will just leave you `print 3` in most languages. There just won't be x, y or z values after the optimizations are done.
If a value is to be calculated then passed down some stack of functions, but during optimization the value is folded into a constant, then the functions rewritten to use the determined constant and then the now unused parameters are dropped, what is it supposed to do? Pretend the value is still there?
Faking stack frame details seems particularly counter to what I want a debugger to be doing while I step through code. Leaving the variables in place would slow and bloat the generated code.
>common operations like printing arrays or memory being hard and obscure
obscure, perhaps. I did have to look up array printing syntax, as I usually only print variables or specific array entries.
It's not hard that I see, however.
$ gdb a.out
(gdb) start hello world this is a test
Temporary breakpoint 1 at 0x4004da
Starting program: /some/path/a.out hello world this is a test
Temporary breakpoint 1, 0x00000000004004da in main ()
(gdb) step
Single stepping until exit from function main,
which has no line number information.
__libc_start_main (main=0x4004d6 <main>, argc=7, argv=0x7fffffffdd58, init=<optimized out>, fini=<optimized out>, rtld_fini= <optimized out>, stack_end=0x7fffffffdd48) at ../csu/libc-start.c:325
325 ../csu/libc-start.c: No such file or directory.
(gdb) print argc
$1 = 7
(gdb) print *argv@argc
$2 = {0x7fffffffe0f0 "/some/path/a.out", 0x7fffffffe10b "hello", 0x7fffffffe111 "world", 0x7fffffffe117 "this", 0x7fffffffe11c "is", 0x7fffffffe11f "a",
0x7fffffffe121 "test"}
(gdb)Yes? The computer is supposed to work hard so that humans don't have to.
> (gdb) print *argv@argc
Interesting! I didn't know that syntax, I've been doing print argv[0], print argv[1], etc all this time.
Also, if you don’t log and someone asks you “what was that yesterday” (not necessarily an error), good luck with your debugger-only approach. Logs are everything - ask your sailor, pilot, vehicle maintenance guy, credit manager.
If I was forbidden to log and was asked how to make a debugger better, I’d say that it should use #preprocessor directives right in modules, in which I could set up watches, object dump conditions, i.e. essentially logging. Please realize that at the end of the day all debugger does is printing something conditionally. But then why not just use plain ifs and prints (and few “if (DEBUG_ENABLE)” in case you want to omit it from the executable).
^ doesn’t matter if it is console or some service, as long as you can see it in real-time
This ignores an awful lot of nuance. If I’m stepping through an algorithm interactively and deciding on the fly what I want to pay attention on a given iteration (or which iteration I want to start paying attention at), or running until a particular thing of interest changes, or any number of debugger scenarios, you can argue that logging all such info all the time and then poring over the output in a log technically gives the same information, but at that point you are positively drowning in data. A debugger gives you the ability to dynamically change what you care about “logging” as your idea of what you need to know changes. I feel like nobody who has ever ever experienced a truly fast (perf-wise) and fluid debugging experience would ever deny how nimble it feels and how you’re never going to get there by deciding ahead of time what is important to log (or by logging absolutely everything).
I also think a lot of people are tainted by the debugging experience in a lot of more modern languages. Golang, python, even Java.. the debugger perf is miserable. Give me msvc 6 and three Fn keys and it feels like you are controlling the processor in real-time.
I use GCC exclusively, and its type-checking of param types vs format string largely prevents me from spending time in gdb; pre-GCC adoption, the biggest source of segfaults in my world was printf type mismatches.
Addendum:
I'm sure it violates every coding convention ever created, but, in almost all cases, once I add a debug printf to my source code, for example,
ED && DBG( "%s", foostr ); // ED is usually a function-scoped enum normally ==0
I leave it there forever. This allows all DBG's to be eternally subject to GCC printf param type build time checking going forward (preventing the debug print code from rotting (as it absolutely would if commented out), at minor maintenance cost (typically only when I switch to a new GCC release)) while not imposing any runtime cost (unless I flip ED to 1 and rebuild to enable function-specific detailed logging).As far as this practice injecting large amounts of comprehension-thwarting noise into my source code: all such `ED && DBG(...);` code sequences are positioned in their own column far to the right of the non-DBG code (needless to say, I don't subject myself to 80-column coding-convention limits; more like 160-column). That this results in multiple statements per line (another typical taboo) is a don't care, since I'm never using a debugger to single-step thru code, and in any case, by convention my DBG() calls never modify program state.
It works great for me, but probably not for anyone else.
No shade. Just wondering.
addr2line 99% of the time can tell you what line of code was a bad boy.
Like you I also leave debug printf's behind. I find that they also function as comments about what's important and likely break.
Unlike everyone else I've seen I have a command line interface I can use to poke at the program while it's running.
I didn't mention earlier that a key component of my "leave DBG printfs in the source code and let the compiler both (a) type-check the trailing param types vs the format string, and (b) elide codegen (executable file footprint) for those same DBG printfs due to the compiler optimizing those calls out" approach is that I build with -O3 which (at least historically; I haven't checked recently) provides a poor gdb debugging experience because the optimizer frequently de-correlates source code lines from generated machine code; if I encounter a segfault, I must suss out a repeatable test case, clean rebuild with -O0 (no optimization), keep fingers crossed that the test case still fails (it usually does), and repro under gdb. The hassle associated with this gyration inhibits my debugger use in non-segfault cases (though these days a clean rebuild is a lot faster than it used to be).
E.g. in Pernosco you can capture most of the benefits of logging by querying for the executions of a particular line, filtering on a conditional expression, and rendering the values of chosen expressions at that line: https://pernos.co/about/expressions/ ... except that you get those results without having to recompile or even rerun your program. And when you find the event you're looking for, you have instant access to the complete program state with the full power of the debugger. replay.io has something similar.
Part of the problem is I often am looking at interactions across systems, so just having a bunch of logs is helpful. Versus having a bunch of debuggers open. Learning the logs also helps me get better at debugging prod: using what's available. But I also add in a bunch of random `console.logs()` too.
I'm looking forward to tracing becoming a more regularly used tool. Being able to see a timeline, across services, is hugely powerful. I hope some-day we design languages that either integrate tracing, or, ascending into high-fantasy, languages where the runtime itself starts to be powered by tracing-like structures, where the tracing also serves like event-sourcing for your galaxy. Innovating on language capabilities, rather than just ever-struggling with language syntax & capabilities & types, feels vastly under-explored.
> especially if you use async logging
If you are async logging then pretty much all of this goes out the window though. Firstly is your formatting async or does it happen when the message is enqueued? Message formatting is expensive, and the longer the message the more expensive it is to format. If you're dealing with low performance devices, you can definitely expect this overhead to adjust the performance and timings of your code. Running printf("%s\n") on a 1MB JSON blobis going to mess everything up, for example. Secondly, assuming that problem is resolved, you still need to order the messages to be printed. There is absolutely no guarantee that two messages sent at the same time from two different threads will be printed in the same order unless you enforce that through some other primitives - at that point see above.
Using print statements
This is Kovid’s favorite way to debug
and he's gotten really far with it. Granted, he is not the average developer.[1] https://manual.calibre-ebook.com/develop.html#using-print-st...
You use what works for you, i will use what works for me. Which is sometimes a print and sometimes a debugger depending on circumstance.
I work in that field so nonsense like that does piss me off. Yes I used logs in embedded and other cases where debuggers weren't an option. But you don't tweet about those cases. This is clearly a stupid thing to say in this case.
The only time I find debuggers useful is if you can tell the issue from the current value of the variables, or if the issue has already been isolated to something like a tight loop of 10-20 lines that are doing a tricky operation.
The biggest issue I have with prints is that I need to recompile, deploy my change and reboot - probably getting towards 10 minutes, whilst with a debugger I can inspect additional state without any work, and some times even do some limited patching to fixup variables or replace something with a NOP.
The easiest way was to simply attach a wire to the interrupt pin in the CPU running to a speaker.
The resultant buzzing was all the info I needed. (Human ears are very good at detecting patterns.)
When using something like JavaScript or PHP where the tooling is finicky and fragile I just give up and use 100% print debugging.
I love that you can hit spacebar on any local variable and "QuickLook" preview it just like a file in the finder. It even renders UIViews, and previews 3D models right in the debugger. The 3D view debugger, memory graph inspector, and all the other tools are just great
Well, if slightly different timings changed behavior, I'd consider it a bug. The microservice systems I usually develop have strong guarantees of processing order via work queues (first in, first out), various stateless components communicate only through channels/queues (shared nothing) and different timings from run to run are absolutely expected because of the unpredictability of the network and external components. The rate of processing events in the queue, when debugging locally, isn't usually too high to "break" the output (i.e. when it's interleaved from multiple workers), even though it happens sometimes, but only if it's in-process. If a thread of execution is its own container instance, they'll have their own stderr output attached, so there's nothing to synchronize.
I understand what you're talking about, something like audio/video processing software or gamedev would suffer from printing as it would mess up the timings. But I'm not sure if stepping with a debugger is any better in that case.
If you're debugging, you already have a bug that you're investigating. Let's say it's a race condition - if you isolate it to a certain function there's no guarantee that adding prints will actually show you the state of what you're computing without the prints.
> But I'm not sure if stepping with a debugger is any better in that case.
No, it's definitely not. There's absolutely no guarantee with a debugger that what you're looking at is the state of the bug either. But saying that you can debug multithreaded code with prints is just false.
Not every bug in multithreaded programming is a race condition bug, I'm not sure why you are equating "debugging a multithreaded program" with "debugging a race condition". If your program consists of stateless shared-nothing workers which take an input and produce an output, most of the state operated on according to business rules is local to the worker and the current request, with no shared state (in memory at least). In my experience, with this kind of setup, most of the bugs are logic bugs and silly programming errors, not race conditions. The problem is that data can be passed from worker to worker between threads, processes, machines, and it's easier to debug it via logging. I'm not saying you should debug race conditions with print(), but a debugger with stepping won't help you either. It's a hard task in general - to debug race conditions, not sure why it's brought up. Printing/logging at least is useful as a first step to investigate what's going on at all.
The exception is with JavaScript in a browser, since the browser is both the native environment where the code is running, and can show the highlighted code and let me set breakpoints. However, even then, more and more of my JS code takes the form of expressions, not procedural statements, and breakpoints just don't help much with those. So in practice I still use them rarely
Would you mind expanding on that?
I might use a stepping debugger on the odd procedural/stateful code, where there are actual variables that change over time, but that's less and less common for me these days
And when I do use a debugger I always just use it to inspect variables; quicker to just print those same variables: less steps, and especially works better if the same code will be run 10 times and I want to print those vars 10 times too.
I never found stepping over code useful in a debugger, and I tried to use it many times because surely it's useful for some, but turns out it's not for me.
Having the problem talk back to the programmer via prints/logging is a perfectly sensible way to do this.
I have successfully used prints to fix tricky issues, in my opinion it is pure snobbery to deride this approach.
One thing I've noticed doing both .NET and Rust at work is that I often do use my Jetbrains debugger for C# but rarely do I use gdb or lldb for my Rust.
Rust feels a lot easier to stick a few more debug logs than it is to use the debugger. C# being VM based means it has a first class debugger that is invaluable to the language (I often joke that our C# code follows Debug Driven Development)
Print debugging just works fine and it's a great aid, and I've seen most of my coworkers make use of it. Debuggers are great for more domain specific usages, people who work on a very specific project or service or application should definitely set up debuggers, but in my day to day work I probably switch between something like 10+ different projects weekly. Most of those also operate in such a distributed/remote manner that it's simply not possible to (easily at least) install a debugger with breakpoints and instrumentation. I've done my fair share of gdb debugging for performance profiling, and that's fine, but that's also a much more involved process that is not as simple as simply writing print("we got here!") all over the code :)
I write tons of fast unittests, so that I can get fast feedback from the system if anything unexpected happens. I think print debugging works quite well for this setup.
"I think the vast majority of developers still debug using print() statements."
— email from a founder
E.g if there's some bug in my search function resulting in weak play, but it only shows up at higher search depths for some reason. The search function is recursive, and the number of nodes visited grows approximately with sqrt(b^d) in the best case with ideal pruning. b is the branching factor(in chess this is ~30) and d is the search depth. That's billions of stack frames to look at. The only way I've found of debugging these is through print statements and grep/awk. As well as programming it to generate some statistics that may or may not be helpful. I've yet to find much use from debuggers.
Usually when I use a debugger it's after I already narrowed the problem down to a specific section and I need to view more state than dbg!() can practically show me.
But since some bugs will only appear in production, and I can't deploy in production a development build, nor can I connect with ease a debugger to a process running in a Kubernetes pod, I have sometimes to use logging for debugging. Those cases are very rare, but they might still happen.
There is typically also both compile-time and run-time log-levels, so that you can disable logging in release builds but still keep all in the source code.
Experienced developers aren't marked by debugging one way or the other, but using the appropriate tool in the appropriate situation.
I also think printf debugging is one of those first to learn last to master sort of methods, as their utility is closely tied to the accuracy of your mental model of the code.
On the other hand, if build/run is a 15-minute cycle, a debugger makes a lot more sense; also for things like apparently random crashes that leave a coredump.
The reason is that the Developer is Reasoning through the Code and Playing Computer simultaneously thus forcing him to make sure that both thought streams are "synchronized" at the stage of the debug print() line statements. It is this upfront work which goes into figuring out what the Developer intended vs. what the Computer really does that is so valuable for Code Comprehension.
Thinking as I write it probably means it's the print statement that should be improved for debugging if anything. Imagine a debugger that collect them, with the memory / variables and stuff. Could also be turned into break points. Alternatively, could a newly developed terminal be coupled with a debugger like that?
Not only does it add more comments to critical parts of the code. It also allows me to enable them briefly in production or staging if needed.
Actually debugging code happens rarely these days as it's usually unnecessary.
PS: Copilot makes writing logs a breeze for me and taught me a trick or two about formatting.
Some people rely heavily on debuggers for their day to day coding. I learnt to code without debuggers or IDEs - there weren't any source level debuggers. Then embedded/microcontrollers where the tooling existed but was very expensive (ICE etc.). I definitely used debuggers more at some point DOS/Windows Borland and MSFT IDEs but mostly as time went by I found that I can usually just figure out the problem without needing a debugger or write correct code in the first place. If you're working on a large distributed system your traditional debuggers don't really help that much any more. But sure, there's situations where you'd still use one.
I remember using watches like decades ago and then I just didn't need to any more...
Whatever works I guess?
Everytime I try to step through, there's all these arbitrary access layers that have to be hit.
It's worth it really.
And maybe in the future bundlers will not be needed anymore. On recent browsers with HTTP2 and gzip compression, I question the need of bundling at all, especially if the dependency tree is limited (yes, that's a HUGE if). With HTTP2 multiplexing / connections kept alive, Good old <script> tags might work quite well now (with type="module").
[1] https://webpack.js.org/loaders/babel-loader/#customize-confi...
print *, 'var_name:', var_name
statement is marshaled up perfectly by pretty much every MPI runtime. There just doesn't exist a user-friendly MPI debugger," he said cunningham's-law-ily.
Think of this as deep introspection into how well your design holds together. If you can't write an effective debugger for the thing you're hatching (or at least, an API and some good tests), then maybe you're looking at a design flaw.
And in any event, you'll have a debugger when you need one. And you will need one. This will probably speed up your development, and your users will hate you less.
But there are too many languages out there in production that don't have debugging support worth a tinker's damn.
(I spent quite a few years working on toolchains, jitters and runtime systems. It's fun stuff).
Python matches up with this style perfectly. The next time you go into insert mode and feel like adding print(x), add a call to breakpoint() instead.
https://docs.python.org/3/library/functions.html#breakpoint
`debugger;` does the same thing with browser JS engines.
- for a simple bug, it can allow to understand the root cause and fix the problem before even reproducing it
- for a tricky bug, it can help understand how to reproduce it, and verify that you are reproducing the correct one - and then comfortably explore it in a debugger.
Source: I had been troubleshooting both kinds of the above in networking gear for a decade as my professional occupation.
If it is weak, annoying, then people use prints
When using C# with Visual Studio and its strong debugging tools, why would you want use prints? (except specific cases that require it)
You can modify code at fly, jump between lines, evaluate expression at fly, use conditional break points
Why bother with some prints?
Print statements are sometimes needed for embedded, but that's probably my least favorite part of embedded.
Lisp live debugging is underrated.
We have so advanced debuggers, OS telemetry, events, that plugging into print statements as first option reveals either lack of tooling or education.
No one wants to learn about all that crap when they already know how to easily output from the software they are developing. I don't want to be debugging the debugger or logger or whatever else.
1/0
…or…
raise “potato”
(it may also be the best choice if you're debugging a language that you hardly know, so you just want to throw an error and halt)
It’s pretty trivial to setup a debugger in most IDE’s these days and share a config with your team.
Print logging falls flat on its face in async systems.
I feel like the ven diagram of people using text editors as a replacement for IDE’s and people doing print logging must be a circle.
it's really fundamental to have good logging! both to discover issues as they unfold, and to debug past runs of the program for which you didn't know there was a problem and as such you didn't run the code in a debugger
The approach you described is much more structured and absolutely vital.
Use with a filtering tool to filter in on a specific area of concern on specific threads.
Bonus if your logging also is tracing and ingested into some big data or telemetry tool.
Printf debugging makes you think about the program, build hypotheses and test them.
Stepping through the program is hit and miss.
If you're ever tried an integrated IDE debugger for say debugging variables, I don't think you can possibly say print debugging is better over one keystroke for setting a breakpoint, one keystroke for running the program to the breakpoint, and one gui window that just displays the local values.
print debugging is better over one keystroke for setting a breakpoint
A breakpoint is by definition a point in your process where you can inspect things. You can’t inspect the entire path without tediously stepping through it. With debug logging you can see the path, but have to pick what you want to see to not overwhelm yourself with output. For me the latter yields much better results in general.
Well these are environments that don't allow nice code editing and debugging. So you've got to work with what you have. Doesn't make it any nicer. Certainly doesn't justify printf debug over other debuggers.
> With debug logging you can see the path, but have to pick what you want to see to not overwhelm yourself with output. For me the latter yields much better results in general.
Are you seriously suggesting that the time and brainpower it takes to type out the code for the prints for the entire code path, AND figuring out when the logic is for the prints happen is better over a debugger stepping through the code path? This is one of the least meaningful things a programmer could do unless they love typing for the sake of it.
It's what I meant by a mediocre norm - you like coding but forget the end goal of it, so you heap lots of unimpactful management layers on top.
log("<sel>:", <sel>)
to the next line automatically. Later I format it as a more meaningful log level/message or <C-/> it out if it was a “silly” loglevel. The ability to program an editor changes how you look at things like that. F3 to list, F2 to saveall, and instantly I see the entire new path of execution, with all insights I’ve thought of at this point. It’s like an interactive debugger, but more interactive in some sense.(I’m also barely typing keywords, e.g. “eafu” means “export async function(|) {|}” and so on.)
Well these are environments that don't allow nice code editing and debugging.
Idk. While that is true for some envs, I’m able to choose at my job. And what I’m usually choosing isn’t an IDE (emphasis on D). Sometimes I have to work with IDE-oriented envs and sure, there I have to use it because console and dump tools suck, but to me it feels like walking through the mud.
Also, sequential console logs are much easier to comprehend than breakpoints when you’re debugging concurrent events.
If I can produce a printf() log/trace + grep, picking out needles in concurrent haystacks is incredibly efficient. 1 in 1000 events are very detectable. Breakpoints would wedge, possibly crash, and frequently mask/alter execution.
And I say this having been "the guy who can debug anything" at a few places using debuggers, having written programmatic debuggers, and built all sorts of contraptions to debug issues as needed.