Hacker News Reader

Comment by jayd16 | Hacker News Reader

> If you are doing anything involving multiple processes, or distribution, debuggers are useless

How so? Can't you just attach to each process? Do you mean at production scale?

This reminded me of a hail-mary use of debuggers in the late 1990s...

Debugging a show-stopper synchronization flaw in the bootstrapping of a parallel job, I had to tell the job system to launch each of 64 nodes wrapped in gdb wrapped in xterm with remote X display back to a laptop. It was something that had "worked in test" reliably, but that was always on a smaller number of nodes or simulating a larger number with time-sharing. It seemed to need real parallel hardware allocations to show up.

So, I was able to launch the job, allow all nodes to run until distributed deadlock, then interrupt and show all thread stacks on one debugger after another until I found the odd process which was out of phase with the rest. As soon as I saw the stacks, I had no further need for the debuggers. Just knowing the "impossible" state configuration happened was the necessary clue.

I have seen lots of satisfactory use of logs for diagnostics, but this is a case where I think the debuggers were almost essential. The debuggers gave a distributed state snapshot due to the deadlock-induced quiescence of the whole system. To have logs show the same snapshot would require a perfect arrangement of unbuffered logging so that the final state would be visible and not caught up in RAM of some or all of the stalled processes.

Sounds like all you really needed something like a signal handler that would dump the stack traces of all the threads.

I've used that a lot black-box debugging java processes in production.

I also wrote such a signal handler for a ruby app.

One thing I have used is drop into the Python debugger (also ipdb) from within a signal handler. Then I can run a kill, and have it drop into the debugger.

There are some of cute hacks to get a remote debugger which I haven't used in years.

https://github.com/sassoftware/epdb can start and connect to a remote debug instance

https://github.com/gotcha/ipdb

In a distributed parallel computing scenario, dumping stacks from 64 processes via a signal handler would have created 64 different files scattered in 64 different remote filesystems. Ironically, the purpose of our library was to bootstrap a distributed communication network and so could have easily supported gathering of remote diagnostic data _if it were not the part that deadlocked_...

But worse, we were debugging our library linked into someone else's application. So, we would have needed their cooperation to add signal handlers. And, we naively thought our library had already been sufficiently tested and did not anticipate the need for last-minute debugging when our user moved their application to a different computing resource that was not available during prior months of preparation and prototypes.

So, the ugly app-in-gdb-in-xterm with X over Internet addressed all that with adhoc instrumentation and communications that could be folded into the existing parallel, distributed job on short notice.

This was also in the days of pthreads in C. The other valuable use of gdb I recall was for memory watchpoints to help track down unexpected changes to certain data. The other tool we got a lot of use out of in those days was Purify to help audit for use of uninitialized memory and for memory leaks.

> In a distributed parallel computing scenario, dumping stacks from 64 processes via a signal handler would have created 64 different files scattered in 64 different remote filesystems.

I was working on a cluster and often needed to do the same thing on all the nodes, run some database query or os command, and gather and integrate the results. I wrote yet another Python stream-objects-instead-of-strings shell (https://geophile.com/osh), which included features for distribution: Run the same command on all nodes, bring back the results, merge streams, gather files, distribute files, etc.

That was several years ago, and it wasn't a full shell. Since then, I've done an improved system that actually is a shell (https://marceltheshell.org, https://geophile.com/marcel).

Even in development.

You could, I suppose, but I'm not sure why you would want to. Going back and forth to two debuggers (at least) seems like torture. And with multiple processes, timing and synchronization issues could make interactive debugging a real nightmare. Why? Why would you do this?

Interactive debugging doesn't scale in any dimension.

It's really not that hard, especially if you can run them all from the same IDE. I can see how it starts to get out hand but it's not at n=2.

Early in my software career, I was debugging a compiler problem. Version X worked, version X+1 did not. I debugged by tracing both step by step, in two gdb sessions, looking for when behavior diverged. Pure torture, and that was an extremely simple case -- the processes were not communicating in any way.

All the usual problems, but doubled. Oops, I stepped too far, start over. Oops, the other one stepped too far, start over. Oops, start over, oops, start over.

Never again.

I used intellij to debug 3 processes over ssh, it worked seamlessly. The problem with logging is that you have to know where to log. Debuggers are excellent for exploration, where logging implies you already put the code. If you forget some log you need to do the whole build/deploy dance, with a debugger I just restart. There are even time travel debuggers that allow replays.

> The problem with logging is that you have to know where to log.

I start with basic lifecycle logging, at an INFO level: The FooBar is created, the FooBar is destroyed, and major states in between. If a FooBar manages a set of things, then I might also add DEBUG level logging for operations on those things. But mostly, detailed logging comes later, as I investigate problems. You really can't anticipate too much logging, because you don't know what's going to need it.

Reply on news.ycombinator.com