Hacker News Reader

Comment by adsharma | Hacker News Reader

I was at Google in 2005 on the other side of the argument. My view back then was simple:

Even if $BIG_COMPANY makes a decision to compile everything with frame pointers, the rest of the community is not. So we'll be stuck fighting an unwinnable argument with a much larger community. Turns out that it was a ~20 year argument.

I ended up writing some patches to make libunwind work for gperftools and maintained libunwind for some number of years as a consequence of that work.

Having moved on to other areas of computing, I'm now a passive observer. But it's fascinating to read history from the other perspective.

View on HN

> So we'll be stuck fighting an unwinnable argument with a much larger community.

In what way would you be stuck? What functional problems does adding frame pointers introduce?

It “wastes” a register when you’re not actively using them. On x86 that can make a big difference, though with the added registers of x86_64 it much less significant.

It caused a problem when building inline assembly heavy code that tried to use all the registers, frame pointer register included.

Right, but I was asking about functional problems (being "stuck"), which sounded like a big issue for the choice.

It's not just the loss of an architectural register, it's also the added cost to the prologue/epilogue. Even on x86_64, it can make a difference, in particular for small functions, which might not be inlined for a variety of reasons.

If your small function is not getting inlined, you should investigate why that is instead of globally breaking performance analysis of your code.

A typical case would be C++ virtual member functions. (They can sometimes be devirtualized, or speculatively partially devirtualized, using LTO+PGO, but there are lots of legitimate cases where they cannot.)

CPUs spend an enormous amount of time waiting for IO and memory, and push/pop and similar are just insanely well optimized. As the article also claims, I would be very surprised to see any effect, unless that more instructions themselves would spill the I-cache.

I've seen around 1-3% on non micro benchmarks, real applications.

Aee also this benchmark from Phoronix [0]:

  Of the 100 tests carried out for this article, when taking the geometric mean of all these benchmarks it equated to about a 14% performance penalty of the software with -O2 compared to when adding -fno-omit-frame-pointer.

I'm not saying these benchmarks or the workloads I've seen are representative of the "real world", but people keep repeating that frame pointers are basically free, which is just not the case.

[0] https://www.phoronix.com/review/fedora-frame-pointer

Wasting a register on comparatively more modern ISA's (PA-RISC 2.0, MIPS64, POWER, aarch64 etc – they are all more modern and have an abundance of general purpose registers) is not a concern.

The actual «wastage» is in having to generate a prologue and an epilogue for each function – 2x instructions to preserve the old frame pointer and set a new one up, and 2x instruction at the point of return – to restore the previous frame pointer.

Generally, it is not a big deal with an exception of a pathological case of a very large number of very small functions calling each other frequently where the extra 4x instructions per each such a function will be filling up the L1 instruction cache «unnessarily».

Those pathological cases are really what inlining is for, with the exception of any tiny recursive functions that can't be tail call optimised.

Yes, inlining (and LTO can take it a notch or two higher) does away with the problem altogether, however the number of projects that default to «-Os» (or even to «-O2») to build a release product is substantial to large.

There is also a significant number of projects that go to great lengths to force-override CFLAGS/CXXFLAGS (usually with «-O2 -g» or even with «-O») or make it extraordinary difficult to change the project's default CFLAGS, for no apparent reasons which eliminates a number of advanced optimisations in builds with default build settings.

I wasn't talking about functional problems. It was a simple observation that big companies were not going to convince Linux distributors to add frame pointers anytime soon and that what those distributors do is relevant.

All of the companies involved believed that they were special and decided to build their own (poorly managed) distribution called "third party code" and having to deal with it was not my best experience working at these companies.

Oh, I just assumed you were talking about Google's Linux distribution and applications it runs on its fleet. I must have mis-assumed. Re-reading... maybe you weren't talking about any builds but just whether or not to oppose kernel and toolchain defaulting to omit frame pointers?

Google didn't have a Linux distribution for a long time (the one everyone used on the desktop was an outdated rpm based distro, we mostly ignored it for development purposes).

What existed was a x86 to x86 cross compilation environment and the libraries involved were manually imported by developers who needed that particular library.

My argument was about the cost of ensuring that those libraries were compiled with frame pointers when much of the open source community was defaulting to omit-fp.

Would it not be easier to patch compilers to always assume the equivalent of -fno-omit-frame-pointer

That was done in 2005. But the task of auditing the supply chain to ensure that every single shared library you ever linked with was compiled a certain way was still hard. Nothing prevented an intern or a new employee from checking in a library without frame pointers into the third-party repo.

In 2024, you'd probably create a "build container" that all developers are required to use to build binaries or pay a linux distributor to build that container.

But cross compilation was the preferred approach back then. So all binaries had a rpath (run time search path to look for shared library) that ignored the distributor supplied libraries.

Having come from a open source background, I found this system hard to digest. But there was a lot of social pressure to work as a bee in a system that thousands of other very competent engineers are using (quite successfully).

I remember briefly talking to a chrome OS related group who were using the "build your own custom distro" approach, before deciding to move to another faang.

> or pay a linux distributor to build that container.

What does this mean?

I didn't mean anything nefarious here :)

Since Google would rather have the best brains in the industry build the next search indexing algorithm or the browser, they didn't have the time to invest human capital into building a better open source friendly dev environment.

A natural alternative is to contract out the work. Linux distributors were good candidates for such contract work.

But the vibe back then was Google could build better alternatives to some of these libraries and therefore bridging the gap between dev experience as an open source dev vs in house software engineer wasn't important.

You could see the same argument play out in git vs monorepo etc, where people take up strong positions on a particular narrow tech topic, whereas the larger issue gets ignored as a result of these belief systems.

You do get occasional regressions. eg. We found an extremely obscure bug involving enabling frame pointers, valgrind, glibc ifuncs and inlining (all at the same time):

https://bugzilla.redhat.com/show_bug.cgi?id=2267598 https://github.com/tukaani-project/xz/commit/82ecc538193b380...

FYI, this "regression" was likely actually a symptom of a purposely-inserted backdoor. See [1] and my comment there [2].

[1]: https://news.ycombinator.com/item?id=39865810

[2]: https://news.ycombinator.com/item?id=39867301

What area?

Reply on news.ycombinator.com