Go has a debugger and it's awesome | Hacker News Reader

Go has a debugger and it's awesome

blog.cloudflare.com

124 pointsFiloSottile69 comments

As someone who has dedicated their career to understanding and debugging software in production[1], it's hard to not be depressed about this -- not only because it (emphatically) isn't "awesome", but also because of the failure to couch this in its proper context: this is -- at best -- a kludge to enable small programs to be debugged in development. While this is fine at some level (debugging is, after all, a good thing), small programs in development aren't where the most pernicious bugs are found; sophisticated tooling should (in my opinion) always focus on the production systems in which the nastiest, most undebuggable problems arise.

Note that this needn't mean that the tools themselves run in production: for example, Valgrind is incredibly useful because it finds really nasty, production-grade issues -- even if it is only meaningfully usable in development. Similarly, postmortem debugging tooling is extraordinarily valuable for production problems -- and requires no tooling in the production environment itself. We have developed such tooling for node.js[2], and found it to be invaluable. There is nascent postmortem debugging support for Go[3], but there is much more to be done -- just hoping that some in the Go community see the value of debugging production problems!

[1] http://www.infoq.com/presentations/Debugging-Production-Syst...

[2] http://www.slideshare.net/bcantrill/node-summit2013

[3] http://www.joyent.com/blog/mdb-support-for-go

> just hoping that some in the Go community see the value of debugging production problems

Some of us do. As I said in another post, Solaris support will only get better. As for Linux, some people obviously care, Go had some form of gdb support from quite early on. Unfortunately, it's not that people don't care about making the gdb support better, it's just that this is all that can be done with the gdb-python interface right now, due to the limitations of gdb...

At least, now with frame pointers, dynamic instrumentation works better.

> just hoping that some in the Go community see the value of debugging production problems!

I just can not remember where I heard it (some podcast maybe), but is my understanding that the Go Team hinted the possibility of something like JMX coming to the Go runtime.

Author here. I noticed before both mdb support and the Delve project that other have mentioned, and I find them great. The point of my excitement for godebug resides in the fact that it's an easy to work with, zero complexity added tool to replace most tedious println() debugging sessions. That is, the non-exciting stuff.

Not all bugs only happen on production load, architecture, etc. Many times (with DNS servers even more so, I'd say) bugs are simple state or logic bugs. The Heisenbugs exist. The complex resource-constrained races exist. And I find it fun to track them down, but many bugs are much more mundane, and godebug is a great tool for catching those.

Speaking instead about hard to trace production bugs, I think the Go design and library deserve their daily share of praise here, since they solve many of those problems. Take Valgrind (or ASAN). Amazing tools. Completely useless in Go -- the language is memory-safe and will make that kind of bugs both hard to introduce and immediately detectable.

And about concurrency, first of all the simple design makes bugs harder to introduce and easier to reason about. But even when they make their way in I find provisions like net/http/pprof and its various profile a pleasure to work with. And finally there's obviously the race detector.

Still, I can't wait to see more tools developed to cover different Go debugging needs. Today I'm just happy about making my println() session much more pleasant and elegant :)

The enthusiasm is fine -- I just wish it were couched in the understanding of the limitations of the technique. I also think it's naive to say that Go's design solves "many [production] problems" -- memory-safe languages solve one set of problems, but often introduce or exacerbate another. (The most obvious example being memory exhaustion: postmortem heap analysis is often much easier on a C-based system than in memory-safe languages.[1][2]) More generally, do not fall into the trap of correctness from smugness: nasty production bugs can exist in any language, and time is much better spent on the production tooling to debug such problems rather than simply asserting that such bugs are tautologically impossible.

[1] http://dtrace.org/blogs/ahl/2004/07/13/number-11-of-20-libum...

[2] https://queue.acm.org/detail.cfm?id=2039361

Edit: added a much better reference.

> postmortem heap analysis is often much easier on a C-based system than in memory-safe languages.

For a specific kind of leak. Java heapdumps and MAT are great postmortem heap-debugging tools (of course, Java Flight Recorder is even awesomer, but unfortunately not part of OpenJDK).

The blog post you referenced in [1] doesn't mention managed language runtimes itself, so let me make sure I understand the point. If I understand your point correctly, it's that with managed language runtimes, particularly mature ones like HotSpot and V8, the C heap isn't used in the typical C way; instead, the runtime has its own heap, closely integrated with the garbage collector. In that case, Go probably has the same problem, just because it tries not to use libc at all. I'm guessing that CPython isn't as bad, because it doesn't use a garbage collector. No idea about Ruby.

Yes -- and you understand the point correctly. And it's independent of whether a language is GC'd or not; to provide memory safety means to provide a layer of indirection between the system's notion of memory and that of the run-time -- and that layer becomes opaque to traditional native tools. I apologize that that reference was a little hasty; [1] is a much, much better one.

[1] https://queue.acm.org/detail.cfm?id=2039361

> And it's independent of whether a language is GC'd or not; to provide memory safety means to provide a layer of indirection between the system's notion of memory and that of the run-time -- and that layer becomes opaque to traditional native tools.

No! You can enforce memory safety at compile time and use the platform's native memory allocation subsystem. There's no need for memory safety to imply any extra runtime support.

As far as I can tell you're right in principle, but even Rust doesn't currently deliver that in practice. With Rust 1.0, at least on Linux, Rust uses its own bundled jemalloc rather than the libc malloc. So IIUC, that means that you can't take advantage of things like libumem, the alternative malloc described in the blog post bcantrill referenced a few comments ago.

Yeah, I figured someone would bring up jemalloc. :) As of a few days ago, Rust allocation is pluggable [1]. jemalloc is considered a "feature" in the Cargo sense which can be turned on and off. (The immediate impetus for this feature work was to use HeapAlloc() on Windows.)

[1]: https://github.com/rust-lang/rust/blob/master/src/liballoc/h...

> (The most obvious example being memory exhaustion: postmortem heap analysis is often much easier on a C-based system than in memory-safe languages.[1])

This is a problem of GC, not of memory safety. Memory safety does not require GC.

> More generally, do not fall into the trap of correctness from smugness: nasty production bugs can exist in any language, and time is much better spent on the production tooling to debug such problems rather than simply asserting that such bugs are tautologically impossible.

The failure modes of the lack of memory safety range from "annoying crash due to null pointer dereference" up to "millions of users at risk of 0-day RCE due to use after free". When the stakes are that high, it's worth spending time to attack those bug classes systematically as opposed to just tackling these bugs in production one-by-one.

Your assertion that "this is a problem of GC, not of memory safety" is incorrect (and I never asserted that GC was the core problem). Memory safety does require a layer of indirection between the native system and the programmer, and that layer introduces opacity to the native system that necessitates its own layer of custom tooling to see through. And memory exhaustion is (obviously!) not a problem restricted to GC'd environments -- any program in any (modern, Turing complete) environment can allocate memory to the point of process death!

> Memory safety does require a layer of indirection between the native system and the programmer, and that layer introduces opacity to the native system that necessitates its own layer of custom tooling to see through.

No, it doesn't. That's literally what Rust (what I worked/work on) is all about. Memory safety is all done at compile time, and it's boiled away to direct access to the platform's native memory subsystem. Platform-native tooling "just works", and I use it every day.

That's terrific, and apologies if you feel I've slighted Rust at all. (I admire Rust's goals to deliver safety without compromising performance or debuggability.) From a native perspective, two things I would love to see with respect to Rust: first, a comment on Alex Light's work[1], as alternative allocators like libumem are often essential for debuggability. Second, I would love to see the applicability of native techniques like postmortem object type identification[2] to Rust. If that "just works" it would be reason alone to seriously evaluate Rust!

[1] http://scialex.github.io/reenix.pdf

[2] http://arxiv.org/pdf/cs/0309037v1.pdf

No worries, and apologies if I seemed snippy. Thanks for the fascinating link on postmortem object type identification; now I have some reading to do :)

See also Debugging at Full Speed http://medianetwork.oracle.com/video/player/3731019771001

Debugging software in production is a hard unsolved problem, the available tools are pretty primitive, and it doesn't help that people often don't plan it into their development and deployment cycle.

We had a client once using our system, who would see anomalies only after long uptimes at sustained high loads. It took six months to find the cause (a race condition in the database code) because we had so few tools at our disposal to look at the post-mortem state of the program. Running the software at those loads for the weeks or months it took to reproduce the problem under a debugger was obviously not a tenable proposition, and reproducing the variability of the particular real-world traffic was difficult to do in a controlled environment. And the software wasn't written with any eye towards making state available for post-mortem debugging.

I guess "don't write complicated multi-threaded software in C" is a pretty good response to the above too. That particular bug couldn't have happened in say Rust (though it could have in Java).

In languages like C it's an unsolved problem. Doesn't mean it's an unsolved problem everywhere (see, e.g. Erlang, which even allows for hot patching without even taking the system down).

Relevant: http://www.flownet.com/gat/jpl-lisp.html (scroll to "Debugging a program running on a $100M piece of hardware that is 100 million miles away")

Yep, old hat in lisp too. Industrial strength VMs tend to do well in this area.

In the QCon talk you referenced in [1], you said that it's impossible to debug a Python or Ruby application from a core file. Why is this? Would an mdb module for Python or Ruby, to delve into the interpreter state and expose it in the debugger, not help or not be feasible?

"Impossible" is a bit strong, but yes, it requires deep and unholy knowledge of the interpreter in the debugger. For an example of what this looks like, see some of the monster block comments in mdb_v8.c[1].

[1] https://github.com/joyent/illumos-joyent/blob/master/usr/src...

My not-so-humble opinion is that valgrind has saved C (and C++) from oblivion, at least for large scale programs.

> postmortem debugging tooling is extraordinarily valuable for production problems

When all else fails this is really all you are going to have to go by. Coming from the Microsoft side of the story I have come to a conclusion that any sufficiently well designed postmortem debugger is a few steps away from being a live/production debugger. Windbg, as a prime example of this, can be used to debug memory dumps, live processes and remote (TCP) processes using exactly the same debugging tools binaries (and scripts/plugins that you write).

Debuggers have to be the biggest source of friction, for me, when adopting things like Go. If you are in charge of "production" you can throw in a few printlns, however, you can't just approach someone like the HP and say "can I put this debug binary on your production system?" No, the absolute best you can hope for is a memory dump and those memory dump had better count as you are only going to get maybe a few.

Maybe it's because not many devs deal with this kind of stuff, which is probably why it isn't a top priority for things like e.g. Go. To some of us, though, the most important language of a feature is easily the quality of the debugger associated with it.

Either way, great slides Bryan - it's nice to see at least one "modern" language taking these issues seriously.

While I understand that this is a fairly young language, it still makes me sad that we seem to erode what's actually cool, what's a good language, what good tools are. It took us two decades to massage the "last" generation of tools, the managed/OO ones, into shape. I've been whining about javas lack of value types, or that java tools didn't allow editing and reloading the debugged code in-place. I have whined for years about how the C# debugger doesn't allow edit-and-continue in 64bit, or how watch expressions can't have lambdas in them.

Meanwhile, outside my windows, it seems the world pretty much just rejected types, debuggers...

"You can do println debugging and generics will just make the language too hard to learn and the runtime too complex".

It used to be the other way around. I was young and the old guys used to say "I'll use my punchcard/command line debugger/whatever, you kids can use your fancy IDE all you want".

Now I sound like a dinosaur for wanting to edit my code in place while debugging, and for being whiny about type systems. What the hell happened?

Also, get off my lawn.

> C# debugger doesn't allow edit-and-continue in 64bit

They added that two years ago:

http://blogs.msdn.com/b/visualstudioalm/archive/2013/06/26/d...

> or how watch expressions can't have lambdas in them.

That's a limitation on how lambdas work. Doubt it could be solved.

That's a limitation on how lambdas work. Doubt it could be solved.

AFAIK, this wasn't a problem with Smalltalk. Watch expressions would have worked in somewhat the same way as conditional breakpoints, and you couldn't have those without allowing Blocks, which are the equivalent of Smalltalk lambdas.

This may be a problem with the CLR, though.

They add watch lambdas in vs2015. Maybe Roslyn enabled it, not sure.

> Doubt it could be solved.

Solved in VS 2015.[1]

[1]: http://blogs.msdn.com/b/visualstudioalm/archive/2014/11/12/s...

There is something to be said for ease of distribution and deployment, as well as playing well with the larger infrastructure/software world. Smalltalk, which was one of the major progenitors of the managed environments you were talking about, didn't do the above. I think that had a lot to do with its downfall.

When your system interacts well with the larger world, this results in more "commerce" with the larger world, which tends to result in greater recognition and larger and healthier communities. It's a virtuous cycle.

Having the best, most well developed developer tools is a win for the individual developer, but fitting in well with the larger world is a win for the entire developer community. The latter is going to be 10X more visible than the former. (Also, think about which communities manage to have both!)

Also, get off my lawn.

Yeah, kids, get off my lawn! At least do enough research to know what's gone before! (Re: Alan Kay's quip about "almost a field.")

Fair enough, but I don't think you addressed the parent's concern.

Most mainstream programming languages fit in well with the larger world, that's why a lot of people use them. They also have working debuggers, a ton of libraries and tools.

Fair enough, but I don't think you addressed the parent's concern.

I placed the concern into a historical context re: language comparison wars.

Most mainstream programming languages fit in well with the larger world, that's why a lot of people use them.

Deployment is a big issue. Go really shines in this regard.

They also have working debuggers, a ton of libraries and tools.

My point is precisely that the 1st and last items above aren't everything.

That's no debugger. That's a debugger: https://github.com/derekparker/delve

If you want to attach to running processes, use delve, it's awesome. For me personally, even with Go's short build times, you can't just deploy a new binary into a running system to see where a problem lies. Sometimes there is just no way around a traditional debugger.

No, this is a debugger

http://plan9.bell-labs.com/sys/doc/acidpaper.html

or on Unix

https://swtch.com/plan9port/man/man1/acid.html

Watch it in action

https://www.youtube.com/watch?v=ZlGFdflPcFo

Acid is great, the best debugger I know, unfortunately the Unix port in plan9port is completely non-functional.

Go is really annoying to debug in acid though, because symbols contain dot characters. Somewhat ironically this was done because Unix debuggers were getting confused by the unicode middle dot character, so the unicode middle dot is replaced in the symbol and debug tables with a regular dot.

But on Plan 9 we should not do this process and we should keep the original middle dot.

cool, it reminds me os DOS's debug.

"Go has a debugger"

The very fact that that is a headline make me think I'll wait a bit before introducing Go into my tech stack.

You can use gdb - https://golang.org/doc/gdb

It's largely accepted that while you can use gdb, it's not really good at debugging go code. from your own link.. "GDB does not understand Go programs well"

https://github.com/derekparker/delve delve as mentioned other places is more of a true go debugger

You can also use mdb[1] though mdb is somewhat more suited to debugging the Go runtime itself rather then user programs.

Mdb support for Go will be significantly improved in the short term.

Also note that since Go has (optional) frame pointers, tools like DTrace, ktap, kprobes and uprobes, etc work much better (almost as well as for C). SystemTap works too, just that it chokes on Go DWARF without a patch. I will probably fix this soon.

[1] https://www.joyent.com/blog/mdb-support-for-go

I think you should ask yourself how all these tech companies that use Go in production ever reached the confidence to use Go in production without a "proper" debugger available.

It is made by Google, and has Plan9 guys on the team, what else do you need?!

The point I'm trying to make is that you can develop and debug Go code perfectly fine without a debugger, as shown by the large-scale deployments of software written in Go. Not adding Go to your tech stack because of that is... not very well thought through, to say the least.

No, it's a very valid concern. It's quite plausible that the absence of a debugger slows down the roll out to production.

Having a debugger is not the only way to debug. Thinking otherwise is not thinking.

I never said it was the only way to debug, just that between a language that only supports `printf` and one that supports `printf` and a debugger, the latter is obviously a more productive environment to work in.

Replace "Go" with "PHP" and ask yourself if such a claim is a strong validation of a technology...'

Yes, a lot of people have gone to production with crappy technologies, this is meaningless. The real question is: could they have done it more easily using a better technology?

> Yes, a lot of people have gone to production with crappy technologies, this is meaningless.

I don't think that's meaningless. I take it to mean that your (our, my) valuation of technologies does not hold the relative importance (or correctness) that it's perceived to hold.

Replace "Go" with "PHP" and ask yourself if such a claim is a strong validation of a technology...'

As a programmer using Go, I think the comparison is spot on -- if you credit Go for having a much cleaner architecture, vastly better design around security, and profoundly well thought out trade offs.

With some work on libraries and tooling, Go could well become the basis of the next PHP, and become the tool of choice for small web projects. If this happened, the world would be a better (in terms of software-sanity, performance, and security) place.

I'm not sure that's a very good argument.

50 years ago there were a ton of companies that were confidently using hex machine code for large systems in production. That doesn't mean things wouldn't have been 100x easier for them with higher level languages and more modern tools.

Of course I can get by without a debugger if I really need to, but that doesn't mean I won't be a lot more productive with one.

If you can't write code without a debugger, then absolutely, wait. I'd straight-up refuse to write C#/Java without a debugger as well - too many layers and obfuscation.

Go, however, is kind of like a memory-safe C...it's almost impossible to actually create super high-level abstractions where you can't tell what the code is doing.

For me, unit tests and the occasional log/print cover 99% of my Go debugging needs.

No no no no. This is not right.

1. If you debug with this, you're debugging different code to what blew up in production for example. Subtle timing issues wiped out instantly and memory ballooning masked etc.

2. If you leave it in for prod, it's going to have masses of call overhead.

You can debug well with assertions, logging and unit tests. There is no need for this. I rarely have to spin up a debugger these days.

Indeed, it's like they've never heard of the term 'Heisenbug'

It's like they know that not all bugs are Heisenbugs.

Well that's a fair comment! It seems strange to me to describe debugging by instrumentation as 'awesome' - 'adequate' seems more appropriate to me. But then I'm not a Go developer so I don't know how the rest of the toolchain compares.

I use de Node.js debugger a lot, I think is very useful when you don't know exactly what to log, and you don't what to start adding logging lines everywhere. You have one assumption but if you are wrong you can check different var, within the debugger step by step. Sometimes I even use the debugger inside a unit test.

The last time I used a debugger was when with VS/C# or IntelliJ/Java, many years ago. For the coding I do now, mostly Go/Python/JS, I find manual debugging techniques much more effective.

> No no no no. This is not right.

It's as worthwhile as the bugs they can find with it. I wish the author would tell us if he found the bug in the dns program.

A debugger that cannot be attached to a living process, that cannot change or execute data dynamically, is not a debugger. Unfortunately, many languages provide "debuggers" that fall into that category.

Well I guess the Go community is a little bit too passionate about any update regarding Go, even if the introduced new thing is subpar compare to other languages, like this one in the post.

I think that the main issue is that debuggers are often dismissed in terms of importance, and many don't see the true value of debugging beyond simple step-by-step execution.

This is not unique to Go: python's several debuggers require cooperation with the inspected process. This is exactly the opposite of a buggy situation. gdb's python support is very limited in terms of debugging capabilities, yet it's the only option you have if you're using an external event loop handler, such as gtk/qt/glib (not so uncommon). I guess I should be grateful I have those at least.

I absolutely agree with you, my problem is that if I bring these things up in an discussion about languages it results in "who are you kidding, we don't need that feature!" sort of comments. Personally I find it hard to do any sort of computation without the ability is inspect what is going inside.

Could a mod please update this to point to the original announcement and not the CloudFlare regurgitation? The Mailgun folks deserve the love for their hard work.

http://blog.mailgun.com/introducing-a-new-cross-platform-deb...

I was just looking into that. Normally we would do so, but that post already had a major thread: https://news.ycombinator.com/item?id=9409423.

Before closing, I'd like to say a few words about the technique of source rewriting in general. It powers many different Go tools, like test coverage, fuzzing and, indeed, debugging. It's made possible primarily by Go’s blazing-fast compiles, and it enables amazing cross-platform tools to be built easily.

Source rewriting is indeed quite powerful. You can use this technique to write a code coverage tool in Smalltalk. (Grab the source code of a method, apply parser-transformations, compile an instrumented version, swap the bytecode of the original method with the instrumented one. Make the instrumented method know how to swap itself back when everything is triggered.) A typical developer can write such a tool and have it become functional in one or two days. An exceptional one can do it in hours.

Iterating more produces more innovative applications and software tools. This is why fast compiles and parser tools are a winning combination.

I also use a well-known debugger called fmt.Println

Wasn't ogle supposed to be the official debugger?

https://godoc.org/golang.org/x/debug/ogle/program

I fundamentally dislike this approach to solving the problem. It feels very crude.

Reply on news.ycombinator.com