Comment by cbanek | Hacker News Reader

Comment by cbanek | Hacker News Reader

Parent Full thread

I never knew that Linus didn't like debuggers, but I completely agree, though for slightly different reasons. I haven't stepped through something in a debugger in probably 10 years.

I don't do kernel development, most of my stuff is distributed systems / networking / devops type stuff. In this realm I feel like debuggers are almost an antipattern because:

1. Distributed systems with a lot of nodes means you might have to attach debuggers to everything, and that's just too hard to control.

2. Anything with networking, like web services, distributed systems, protocols, etc. will likely time-out the connections while you step through line by line. Once your TCP socket gets closed on you from below or the other side timing out waiting for a response, then you have to set up your debugging environment all over again.

3. Most of the real hard bugs I have to deal with usually involve timing or race conditions. This means that I might not be able to reproduce them in a debugger, and sometimes even the act of attaching a debugger makes the timing different enough to not repro the bug.

4. In production of these systems, most times you can't just hook up a debugger and basically shut down the system while you try to diagnose a bug. I collect what info I can from the logs and move on. In this situation, you need great logs. Great/useful logs are built over time, by adding log statements to the right places. So for every bug I diagnose, I keep the logging statements in I used for debugging (although I usually don't leave them emitting messages, because I have log levels).

I worked on Browser devtools. The #1 used tool was console. Nothing beats being able to put some console.log and see its output. Console.warn is even better since it captures the stacktrace.

Logs give a timeline of change.

I had this idea of console.snap but never got around to it. The idea is that it would not only capture the stacktrace, but a shallow copy of scopes at every function. You can query the snap to see how a variable changed, or find what set of variables caused something else to change later on.

I feel like this is the holy grail of debugging. Smarter logs that you can reason with and do time travel analysis.

I never got around to doing it, but now working at an analytics company, I see it being very valuable as it saves so much guess work.

I totally agree - I love the ability in python to print out the stack trace, I use it all the time! I would love if that also had variables as well (although, this is tricky, because you have to worry about cycles between the variables if you are trying to print everything out. Maybe a list of variables to print out with the stack trace would be enough?)

That’s why I suggested a shallow copy

> 3. Most of the real hard bugs I have to deal with usually involve timing or race conditions. This means that I might not be able to reproduce them in a debugger, and sometimes even the act of attaching a debugger makes the timing different enough to not repro the bug.

Debuggers actually are often really good for that sort of thing. If you can break the code when the race happens and start looking at the stack and execution trace finding what happened is usually trivial. The big deal is by understanding very clearly what happened you know your fix actually fixes the problem.

I'm talking about race and timing conditions between multiple machines. It's a lot easier to just log the various points to find the ordering of things for these kinds of conditions. As long as your clocks are closely sync'd, it's easy to merge the logs together to look and see what multiple machines are doing as a call flow.

Of course understanding the root cause is the key to fixing any issue.

Reply on news.ycombinator.com