Hacker News Reader

Comment by benesch | Hacker News Reader

The Go language literally requires that pclntab be included in release builds. I'm with you—it seems kind of crazy that this was designed into the language—but there you have it.

The reason is that Go's standard library provides functions that allow retrieving a backtrace and symbolicating that backtrace at runtime:

* https://golang.org/pkg/runtime/?m=all#Callers

* https://golang.org/pkg/runtime/?m=all#CallersFrames

Unlike in C or C++, where you can link in something like libbacktrace [0] to get a best-effort backtrace, those Go functions are guaranteed to be correct, even when functions have been inlined. This is no small feat, and indeed programs compiled with gccgo will often be incorrect because libbacktrace doesn't always get things right when functions have been inlined.

[0]: https://github.com/ianlancetaylor/libbacktrace

View on HN

Is it that common for these functions to be used? Perhaps transitively through some popular libraries? Just being in the standard library doesn't even necessarily mean that these functions (and the data they need) should get included by the linker.

Any program that uses a logging framework, including the stdlib log package, will wind up depending on runtime.Callers at least transitively. That’s probably most Go programs; certainly most of the programs large enough to be worrying about binary size.

Unlike in C, there are no macros like __FILE__ and __LINE__, so there is no alternative to runtime.Callers (short of preprocessing your Go source code).

You can still get a backtrace without symbols though.

Why couldn't the Go team introduce a flag that strips symbols, while making clear to people that they should only use it if they are okay with backtraces looking like

  #1  0x00007ffff7ddb899 in __GI_abort () at abort.c:79
  #2  0x0000555555555156 in ?? ()
  #3  0x0000555555555168 in ?? ()
  #4  0x000055555555517d in ?? ()
  #5  0x0000555555555192 in ?? ()

or similar

Because one of the mantra of Go is not telling users to have a "debug" build and a "release" build. The development build is the one that goes into production, with no difference in optimizations, symbols and whatnot. This has pros and cons, like all tradeoffs.

Thanks; I didn’t know that, but it makes sense. Not a design decision I agree with, but it’s coherent, at least.

It seems like this can only be a con if other languages give you a choice. In other words, in C++ you can decide that your debug and release are the same, or you can make them different if you know what you're doing (like a good IDE does).

Having more options is inherently better in C++, a language designed to reward mastery. Having fewer choices is better in Go, a language designed to make onboarding a new developer easy.

For any given tool you're going to spend most of your career at the top of its learning curve, so it doesn't make sense to weaken the tool to optimize for the brief period when you weren't.

As a developer I agree with you, but I think if I were a FAANG I'd be thinking: developers are crazy expensive, how can I commodify them? Every C++ developer is unique. There's endless variety of language subsets and build situations. It takes enormous investment to bring someone up to speed on a project, and only mandating a heavily-restricted subset of the language can make teaching it to fresh college grads practical. These are exactly the things Go is designed to avoid.

In my experience, new developers are not normally charged with determining the details of build flags, deployment systems, etc.

Not to be too judgmental, because I recognize that every organization has its own way, but that sounds like a sign of organizational disfunction.

I just mean, when you join an existing project, most of that would already be set up.

In every project I’ve been on, I was asked to ramp up in the first few weeks by fixing small bugs here and there in the core codebase, not by setting up CI/makefiles/build infra/etc...

I actually think I might be totally missing your point, since I don’t really understand what you mean or how you arrived at the conclusion that it’s a sign of dysfunction.

I hope that's how it would work for me- getting the task of setting up makefiles on the first day- before I know anything about the codebase- would feel pretty daunting, to say the least.

That's bullshit. C is simple and you have masterpieces written on it. And 6502 ASM is even more simple and yet some C64 games are a pure wizardry.

The "26M pclntab" issue sounds like an excellent argument against that design decision.

Counterpoint: our work infrastructure for symbolizing stripped binary stacks sucks. I'd love to get backtraces with symbols from the field.

But also this entire section could be compressed and seekable and only expanded on demand. No need to leave it as a bloated uncompressed blob.

> our work infrastructure for symbolizing stripped binary stacks sucks. I'd love to get backtraces with symbols from the field.

If stripped binaries don't work for you, nobody is saying you should be forced to use them. You would presumably just not avail yourself of the option.

(As an aside -- what's wrong with your symbolification infrastructure at work? This is something I spent significant time working on in a past job, so I'm always curious how other people are doing it).

My employer's internal infrastructure for this just sucks; it's not endemic to stripping binaries.

Specifics? The automation to associate cores with specific builds is bad or non-existent; the automation to load up GDB with the right files and corresponding sources or source branch is bad/non-existent; the fileserver storing symbol data is slow and sometimes remote to the developer across a very thin pipe; etc, etc. All to some extent foot-shooting by my organization. But we have TBs and TBs of build artifacts and hundreds of unique daemons producing cores and also have to debug kernel cores and multiple product branches.

Part of the pain is perhaps that we're cross-building for an embedded FreeBSD-derived system, but the majority of our developers use Linux or Mac, so we can't necessarily use host-native (or host-native-only) tools.

I think basically the situation is begging for someone familiar with the problem-space to sit down and bang out a FUSE filesystem or two (e.g. fetching sources on-demand as GDB reads them, instead of checking out a full multi-GB repo copy) along with a shell or Python script to load up a core. But no one has done it yet and management doesn't really prioritize developer tools.

Why? What problems is it causing you in practice?

Are you sure this is true? Doesn't delve for example build Go source with special flags (gcflags=all='-N -l')to generate debugging symbols. I also remember having to build Go code with those flags for Stackdriver to get the correct debugging information without any optimisations.

This shouldn't be necessary anymore.

Thanks for the reply. Would appreciate if you could expand on this a little, since when? FWITW Google Stackdriver documentation [1] still states that you have to build source with those flags for version 1.1>.

[1]https://cloud.google.com/debugger/docs/setup/go

Backtrace can be amazing at run time if your going to put that info into logs. It makes finding "fringe" errors a lot less painful or more easily reproducible.

Java and .NET can also get backtrace at runtime. Can't remember if they are symbolized or not but i think they are. How do they handle this?

They generate unwind information on the fly as they compile the code which is then stashed to one side, compressed and accessed only if a true language-level stack trace is required. Also they don't expose argument or return values in stack traces.

They’re interpreted/JIT’d and not compiled like golang.

They, have release and debug build modes.

.NET has AOT support since ever via NGEN, followed by Mono AOT, Xamarin,IL2CPP, Bartok, .NET Native and a couple of less know projects.

On Java side AOT has been always supported by JVMs for embedded deployments like Aonix (now part of PTC, PTC, Aicas, ExcelsiorJET, IBM J9.

And nowadays there is SubstrateVM as well, renamed recently as Graal Native Image, part of OpenJDK since Java 11.

Not entirely true with regards to the question.

Yes, Java is interpreted and JITed, but on the bytecode level. When compiling sources to bytecode, line number information is written to class files.

libunwind[1][2] should unwind correctly if a compiler emits correct DWARF CFI unwind table (required by C++ w/ exceptions; available for C in gcc/clang with -funwind-tables).

[1]: https://github.com/llvm-mirror/libunwind

[2]: https://www.nongnu.org/libunwind/

I don’t think libunwind gives you function names if you strip symbols and don’t have debugging information on hand.

Sure. libunwind is taking the place of pclntab here, not the symbol table. Go must also not strip symbols in order to print backtraces; if you want the same thing in C, don't strip the symbol table there either.

Reply on news.ycombinator.com