A cross-platform debugger for Go | Hacker News Reader

A cross-platform debugger for Go

blog.mailgun.com

379 pointsjschlatter62 comments

There's a more, er... traditional, debugger for Golang in the works: https://github.com/derekparker/delve

> When I started this project in January, gdb failed on every program I tried it on. delve didn’t work on OS X, and print-statement-debugging was too slow and limited. What's a developer to do? Make my own debugger, of course.

It seems like he tried to use delve and it wasn't portable at the time. Sure he could have worked on making delve portable instead, but he probably learned a lot more poking around the internals himself.

Does Delve only allow you set breakpoints while running the program in Delve?

Yes: https://github.com/derekparker/delve#breakpoints

Comparing the Linux and OSX instructions is fun:

https://github.com/derekparker/delve#building

If I remember correctly I had to follow the same instructions for installing gdb too :/

Really? I just did brew install gdb

https://sourceware.org/gdb/wiki/BuildingOnDarwin

Duly noted. Thanks.

This looks really great. When I saw how Go does coverage testing using the same approach I felt that the code generation approach would work for a debugger. I'm glad someone did it - and from the few moments of playing with it it seems to be working great. I'm adding this to my toolkit for sure.

This demonstrates the power of Go generators. We're seeing what are typically language features such as generics and debugging being implemented before runtime using code generation. The result is more functionality without an overall increase in runtime size or decrease in runtime performance.

It's unusual and it's opinionated, but it works.

To be fair, this does increase runtime size and decrease performance, but that's a necessary cost that comes with the debugging capability

"that's a necessary cost that comes with the debugging capability"

I'm doing some research that shows this isn't the case. You can have a system where enabling the debugger has zero impact on runtime performance, via dynamic deoptimization.

http://www.lifl.fr/dyla14/papers/dyla14-3-Debugging_at_Full_...

> This demonstrates the power of Go generators. We're seeing what are typically language features such as generics and debugging being implemented before runtime using code generation. The result is more functionality without an increase in runtime size or a decrease in performance.

Monomorphization is a bog-standard implementation technique for generics. It's not at all unusual.

source?

Just off the top of my head, C++, .NET (via a JIT), Rust, D, MLton, and (I think) Swift all do it.

I was hoping for some article or wikipedia entry that would tie your comment with the parent's comment, and with the original article. Monomorphization is, to my understanding, taking a polymorphic function and instantiating it to the specific type requested. This is done at compile time. Now, the tool in this article interleaves debugging code with original source code. So the connection is... both monomorphization of generics/templates and this tool operate by emitting source code prior to compilation?

The original comment was about how both debugging and generics are being implemented with code generation before runtime in Go and described this as "unusual." The response was that this precise technique is how generics are implemented in many languages already; in fact, CFront (the original C++ compiler) was entirely implemented as code generation on top of C. The response specifically discussed generics, not debugging, so I'm not sure why you thought it was talking about debugging.

What is the difference between the generators in Python vs Go?

They're not the same thing. This is about code generators (https://blog.golang.org/generate) while in Python context you're most likely talking about iteration.

Jeremy Schlatter, godebug creator, did an interesting talk introducing the project March 18th at GoSF. The slides are available here: https://github.com/jeremyschlatter/godebug-talk

Norman Ramsey and David R. Hanson implemented this sort of compiler-assisted debugging for C back in 1992, in a debugger called ldb: http://www.cs.tufts.edu/~nr/pubs/retargetable-abstract.html

What do you do for inlining + code movement, where functions become interleaved (as do lines)?

In particular, after inlining, how are you guaranteeing statements from a given function don't get moved before the inlined enterfunction call (and similarly with lines)

Or do you not expect it to ever be able to report right answers for optimized programs? (which is a valid way to live life, of course, but ...)

Since it's modifying the source before compiling it, I expect that the compiler will conclude that most optimizations can't be applied when they cross breakpoint boundaries.

So, when debugging is turned on, the code would return the right answers, but it wouldn't have the same performance.

"Since it's modifying the source before compiling it, I expect that the compiler will conclude that most optimizations can't be applied when they cross breakpoint boundaries."

While true, this depends on the compiler knowing this is a magical breakpoint barrier it can't move things across. The compiler has no idea this is a magical barrier unless something has told it it's a magical barrier. Looking at godebug library, i don't see this being the case, it looks like it translates into an atomic store and an atomic load to some variables, and then a function call, which the compiler is definitely not going to see as a "nothing can move across" barrier.

(Also, having the debugging library alter the semantics of the program is 100% guaranteed to lead to bugs that are not visible when using the library, etc)

"Also, having the debugging library alter the semantics of the program is 100% guaranteed to lead to bugs that are not visible when using the library, etc"

Can you give an example of the kind of bug you expect to see?

say you have:

    x = 1
    x = 2
    bar()

You set a breakpoint on the call to bar and examine the value of x. You would expect it to be 2, but what if the compiler had decided to move the allocation of x = 2 to after the call to bar? There's no reason why it shouldn't. You'd then see x = 1, which would confuse you.

Thanks for the concrete example. The transformed source code that would get compiled for this example looks like:

  scope.Declare("x", &x)
  godebug.Line(ctx, scope, 3)
  x = 1
  godebug.Line(ctx, scope, 4)
  x = 2
  godebug.Line(ctx, scope, 5)
  bar()

The value of x is visible to all of the godebug.Line calls, so the compiler should know that it can't move x = 2 to after the call to bar.

Right, but now you have the opposite problem. Let's say that before the compiler could move x=2 after bar (let's assume bar does not touch x).

Now, in your world, it can't.

So before, you would have seen x=1 in the call to bar, and now when you use the debug library, you will see x=2.

But it's not really material, because bar doesn't touch x (and if it did, this problem wouldn't exist to begin with).

In other words, it might be a bit strange and cause a slight detour in your quest to discover the cause of a bug, but it wouldn't actually change any behavior or cause any issues.

Your argument is essentially: The barriers you insert, even if they block optimization, will never block optimization in a way that changes behavior. This is demonstrably false, since you are, among other things, taking the address of a variable, which means escape analysis won't do things to it, etc.

You can argue "The behavior it changes doesn't matter". As i've shown, 1. it does in a threaded environment (like, you know, go) 2. It depends on whether your code is buggy or not.

IT's certainly true that it never, on it's own, causes bugs. But as i've shown, it can make bugs appear to come or go.

If you don't think that will ever happen, i don't know what to tell you, other than "It has happened in literally every compiler that has ever had barriers like this".

Without any evidence why Go should be different here, i don't see why go will be different here.

Sure, i'll stick to bugs invisible with the library, and visible without it:

If you've inserted compiler barriers it can't move code across, the compiler will no longer perform the same optimizations with and without your debug library.

Those optimizations often make bugs visible, because variables no longer have the value you expect them to at the time you expect it, etc.

Add in threads, and the problem gets worse:

http://preshing.com/20120515/memory-reordering-caught-in-the...

Look at the behavior this has with and without the barrier. It's buggy without it. With it, it's fine.

This is exactly equivalent to:

Without the debugging library, the code is clearly buggy and doesn't work in user-visible ways.

With the debugging library, if i insert a breakpoint where the current asm barrier is, it now behaves correctly 100% of the time, even though it's broken.

I think you're right that it's going to introduce bugs in concurrent code. For example, it's legal to send a pointer through a channel as a way of transferring ownership and never access the object again. If the debugger rewrites the code so that "never accesses it again" is no longer true, it's created a data race.

On the other hand, godebug generates straightforward single-threaded code that creates pointers to locals in a shadow data structure and accesses them later. There's no reason it shouldn't work if you're not using goroutines.

In particular, a previous call to godebug.Declare("x", &x) will add a pointer to what was previously a local variable to a data structure. This effectively moves all locals to a heap representation of the goroutine's stack, to be accessed later. It's going to kill performance, but it's legal to do.

"It's going to kill performance, but it's legal to do."

Sure, but it's going to cause the optimizer to do different things than it would have to that variable. As I said, this essentially changes what the compiler is allowed to do, and will expose or hide bugs (usually hide if it hurts the optimizer) :)

One only has to look at the bugzilla's of gcc and llvm to discover all sorts of fun things that barriers hide/expose.

It's clearly not useful for tracking down compiler bugs, but might still be useful for more ordinary bugs in single-threaded user code.

Good question. I think https://news.ycombinator.com/item?id=9411158 is right, but I haven't looked at it closely yet. Thanks for bringing it up!

What an awesome idea to use gopherjs. Just added it to http://golangtoolbox.com/ - if you have more tools that are great please add them here.

Thanks! gopherjs is a really great project. This was my first time using it and I was impressed.

I've considered putting it up as a permanent playground like http://play.golang.org, where you can debug little Go snippets on the web. Is that something that you would find useful?

I agree with you about gopherjs being an awesome project. And I would love to see a playground-like debugger that you could run directly in your browser for quick debugging!

Hi, not relevant to the topic. I visited your site, specifically "Homepage > Show all categories > Crypto" (http://golangtoolbox.com/categories/crypto) and was met with:

"template: views/layout:views/categories/show:1:301: executing "views/categories/show::main" at <.CurrentUser.Id>: nil pointer evaluating interface {}.Id"

Happens when viewing any category. Looks like a useful site though.

While I totally applaud this effort, how does it not lie? It inserts code into the actual source claiming it is on a line in the original file, but it is in a different line in the compiled version. That seems like it could be a problem at some point, and despite the serious utility in this, I'm not 100% convinced it is the right tool for the job.

Author here. I agree that this could cause surprises. Are there specific cases you have in mind?

Two cases I have thought of are stack traces and logging that inspects the stack. In the former case, if you get a stack trace while debugging it will not mean much because the lines do not match those of your code. In the latter case, logging statements may print the wrong things if they depend on being called a certain number of stack frames below user code. glog does this, for example[1]. I don't have solutions to either case yet, but I have some ideas of how to start. I think the utility the tool provides is well worth those two issues, and I'm hopeful that both can be fixed.

[1] https://github.com/golang/glog/blob/master/glog.go#L536

A neat trick in some languages is to modify the source so that the line numbers don't change, for example by using semicolons instead of newlines and making sure a newline never appears in inserted instrumentation code. But I'm not sure if that's possible in Go.

Go has special comments for that: "//line filename:line"

Thanks, I didn't know that. The go/token docs don't say much about it. Do you know if there is better documentation somewhere?

http://golang.org/cmd/gc/

I tried using this facility a few months ago based on this very short documentation but couldn't get it working, nor could I search out any further doco on it, so because it wasn't really important I gave up. Can I ask... have any of you ever successfully used this?

Yes, me. It works. :) Also it's used by "go test -cover".

I love the in-browser demo of this Go program via GopherJS. So easy to try and become familiar. Well done!

I wrote a similar instrumenting debugger for JavaScript awhile back, but haven't maintained it: https://github.com/tlrobinson/xebug

At one point even I implemented the V8 debugger protocol so you could use the Web Inspector, and an HTTP proxy that would automatically instrument JavaScript files (using synchronous XMLHttpRequests to block execution...), but I don't think I ever posted that code :(

This is really cool, thanks.

How does this handle go routines? Can only one be paused at a time?

Author here. Yes, only one goroutine is paused at a time. I'm planning to add commands to show and jump between goroutines. Still sketching those out, though. Do you think it would be helpful to have multiple goroutines paused by the debugger at once?

I'd like to be able to completely pause all goroutines and slowly analyze the state of each one. You can probably just add some mutex or waitgroup to your line() method and just lock it when a goroutine pauses, no?

Yeah, that should be relatively easy to do. We would just skip the "am I the goroutine under the debugger" check here [1] and add some kind of channel communication here [2]. The hardest thing is probably just the UI. How should that functionality work? I'm imagining something like this:

  >>> pause all
  All goroutines paused
  >>> show goroutines
  1: foo.go:16
  2: foo.go:24
  3. bar.go:10 [current]
  >>> next
  -> // some code from goroutine 3
  >>> goroutine 2
  Now tracing goroutine 2. Current location:
  /*
     Code listing from goroutine 2's current location
  */
  >>> next
  -> // some code from goroutine 2

Is this the kind of interface you were imagining?

[1] https://github.com/mailgun/godebug/blob/5c173f56b398bc13fd41... [2] https://github.com/mailgun/godebug/blob/5c173f56b398bc13fd41...

EDIT: formatting

Yep, something like gdb's threads interface.

In a well behaved program, all theads should probably be synchronized by mutexes/channels etc, so pausing one thread, should in theory pause all related ones.

I think one is fine for now.

It would be nice to have something like this for Rust.

Curious.. I used debugging in LiteIDE a while back and it seemed to work well. Anyone else using it successfully?

LiteIDE was probably using gdb as a backend. gdb is not a very reliable debugger as of the last few Go releases. Here's a demo of a common failure mode of gdb on Go: occasionally running the entire program when you try to take a single step https://youtu.be/qo4RAPxCTa0?t=351

This failure mode is a large part of why I decided to write godebug.

Reply on news.ycombinator.com