Hacker News Reader

Comment by saagarjha | Hacker News Reader

Having debugged a lot of optimized code I would strongly recommend against it unless you are in a context where performance of your build is paramount (games?) or you cannot reproduce the bug when compiled without optimizations. Compilers really do a terrible job at preserving useful debug info when you turn them on. It’s a massive pain to have everything be marked as “optimized out” and reassemble the things you want from other variables or by using a disassembler to manually track which register the value is hiding in.

View on HN

It's not only games. Anything sizeable that needs to run to repro will crumble under 20-100x times slowdown. Multithreaded behaviors will be just different. All those wonderful templated abstractions do not come for free in -O0. Ranges are especially egregious example. Debug build is truly dead outside of unit testing (imho).

Realistic scenario that gamedev uses: deoptimize translation units you are interested in finding or reproducing bugs.

> Realistic scenario that gamedev uses: deoptimize translation units you are interested in finding or reproducing bugs.

Yep, looks like that’s this bullet point:

https://dhashe.com/category/blog.html#partition-your-tus-int...

> 20-100x

This would be really unusual though right? For reference, in plain C code I see a 2x slowdown, in Zig a 4x slowdown (which I still need to investigate why exactly that's the case), and in C++ (even with heavy stdlib usage and on MSVC) at most 10x - which is the absolute worst case I've seen yet. My C++ info is a bit outdated though, have things gotten much worse in "modern" C++?

Or in other words: if you see a slowdown of 100x in debug mode, I would be really concerned about why the performance is so heavily dependent on the optimizer doing it's thing and would start investigating what's the reason for such a massive slowdown.

At work, I have a large C++ codebase in which I can observe slowdowns of one order of magnitude or more with -O0 compared to -O3. And for many bugs it still takes 10 minutes of runtime to hit them, so not using optimizations just isn't tenable.

As far as I can tell, this is worse than for the average C++ codebase. I have some ideas for why the optimizer is able to achieve such large improvements with this particular code and some of that could surely be done with better source, but it's a huge code base and a refactoring on that scale just isn't going to happen.

At least with MSVC if you use SIMD heavily the debug build will be 100x slower easily. I believe it creates debugable SIMD by storing each operation to memory, though I'm not entirely sure how they managed such a massive slowdown even so:)

Anyway the way I deal with it is to mark functions I want to debug with a macro that disables optimizations just for that function.

Does memory of such C++ that can be used in debug build exists somewhere? Yeah, one can still write that. Example from open source:

    tracy_force_inline void swap( Vector& other )
    {
        uint8_t tmp[sizeof( Vector<T> )];
        memcpy( (char*)tmp, &other, sizeof( Vector<T> ) );
        memcpy( (char*)&other, this, sizeof( Vector<T> ) );
        memcpy( (char*)this, tmp, sizeof( Vector<T> ) );
    }

This is quite fast in debug.

Will people who write C++ in 202x gamedev do it? Nah, I don't think so. Just stop even maintaining that pesky 'Debug' build and it will be fine. Build configurations grow in other direction. There are optimized configurations already that do not need local build step - LTO/PGO and friends. People just do not build that locally and real world performance is there.

YMMV of course.

> It’s a massive pain to have everything be marked as “optimized out” and reassemble the things you want from other variables or by using a disassembler to manually track which register the value is hiding in.

If you've got a time traveling / reversible debugger than you can (sometimes) go back to a point where the value was being written / used, at which point it'll often reappear in scope and be accessible.

I believe DWARF's built-in virtual machine should be able to recompute missing values in many cases but I don't think compilers are great at putting the relevant info in, even where it should be possible to compute the right value fairly easily.

The other trick I've found really helpful for "optimized out" values is to find places where they cross boundaries that block optimizations (e.g. procedure calls to another translation unit, so long as you're not doing some kind of link-time optimisation).

e.g. if the value you're interested in is being passed to / returned from a function then inspecting it around the call / return site should have the value available.

Two particular notes around this (exact commands assuming gdb, but other debuggers should have equivalents):

Pass various arguments to `backtrace` rather than just relying the default. Chances are it will have some non-optimized-out variables, which you can use to figure out what's going on.

Use `info registers` and see what looks like a pointer, then cast it to a type you suspect it is. Note that this can be done for any stack frame.

Even with LTO or inside translation units, compilers are (sadly) extremely conservative about changing function boundaries.

I've seen gcc do an enthusiastic job on `static` functions within a C compilation unit - specifically where they only have one call site and so can be fully inlined into the caller.

In that case, the code did become pretty hard to debug due to the extensive inlining and reordering it had allowed. Unfortunate because the only reason such functions exist is to make the structure of the code more apparent!

Maybe that's an exception (and / or maybe it's easier with C than for C++).

Maybe the less is that it's still always worth trying function call boundaries, in case the compiler has been conservative!

Yes, inlining is one exception. Function variants for constant propagation are another. But both of those have been around for a long time and if neither of them apply compilers won't even reorder arguments where that makes sense or deviate from the platform calling convention based on register pressure inside the functions or drop unused arguments or reorder arguments if that improves things. Feels like in general internal function boundaries in the code should be just a hint to and optimizing compiler but somehow there has been very little development in that area. Similar for struct/class layout, compilers won't touch that even if they can see all uses.

feels strange still to see complaints about debugging in production being inconvenient when we should have caught these issues in test/staging. secondly I think not having debug tools and debug data in production is a security feature.

You’re trolling, right?

Reply on news.ycombinator.com