Debugging a Crash in OpenRCT2 | Hacker News Reader

Debugging a Crash in OpenRCT2

voidstar.tech

79 pointselvis7017 comments

You can have debug symbols even in release builds. With gcc, `-ggdb` for debug symbols and `-O2` for optimization can be selected in dependently of each other. Optimized binaries are a bit harder to debug, but locating the source of this crash should be easy even in optimized versions.

As additional complication, debug symbols are often removed from the binary post-build to reduce binary size. The `strip` utility either discards debug symbols entirely, or it puts them in a separate folder as .dbgsym filses. See the gdb `debug-file-directory` option and the `add-symbol-file` command.

I would've put the binary into Ghidra, found the asm by EIP/RIP when the author found the segfault breakpoint in gdb or whatever, then looked for XREFs. That probably would've helped find where the memory/variable/code was. I'm not 100% sure that could've been traced back release -> debug build?

I take it the author didn't know this method, or... I might be missing something as to why that wasn't done?

Coming from a reverse engineering background and then going into regular sweng, I find myself wanting to grab IDA/x64dbg instead of my IDE's debugger very often...

Sadly workplace politics prevent me from doing that and somehow everyone else is productive enough, so I'm just slowly getting used to living in Visual Studio.

what does it say about me that I go for printf()/println!()/console.log before an actual debugger? does it reveal that I'm basically a giant noob? lol

Nothing! Use the tools you have available to you, and pick whichever ones you're productive in. The only thing you should really look into is figuring out what each tool can offer you and being familiar with them, and then making an informed decision when you reach for a tool.

There's a large overlap between experienced debuggers and reverse engineers, because the tracing of a broken program is really a reverse engineering problem in disguise.

I wonder if the bug was present in the initial game, or if the boolean test was accidentally inverted during reverse engineering.

Sounds like a refactor error, from the linked Issue: https://github.com/OpenRCT2/OpenRCT2/issues/16572#issuecomme...

Would Valgrind have found this nearly instantly?

In theory, yes. But in practice, it probably would be hard to reproduce the bug while running 30x slower than normal. On the other hand, maybe computers today are fast enough to run OpenRCT2 under valgrind-memcheck?

I wonder if compiling the game with ASAN would have helped here, saving a lot of manual digging for out of bounds accesses.

That's already covered in the article:

> Assuming the bad index was always present, I thought I might catch it by instrumenting accesses to global arrays. Using ASan was a possibilty, but I believed enabling checking globally on a program full of unrelated memory issues would be too noisy to be of use.

> I believed enabling checking globally on a program full of unrelated memory issues would be too noisy to be of use.

I generally think that all the issues ASan reports are worth fixing. And for a mature program that's had many kinks worked out, there's probably a not-too-enormous number of memory issues. And if there were they probably all still deserve fixing.

> Games often prefer targeted debug features for specific systems or types that are toggled by their own compile-time switches and don't depend on individual compiler extensions.

Other debug features worth considering are listed below. Not all of them depend on compiler features. Some are more security oriented but can end up identifying functionality bugs along the way.

    -D_GLIBCXX_DEBUG
    -D_FORTIFY_SOURCE=1 [1]
    -D_LIBCPP_DEBUG [2]
    -fstack-protector [3]

Of course, I'd probably still start with ASan/UBSan/TSan anyways. But these are good, too. Especially since this is a C++ project, the C++ library debug modes are handy.

[1] https://www.redhat.com/en/blog/enhance-application-security-...

[2] https://releases.llvm.org/14.0.0/projects/libcxx/docs/Design...

[3] https://developers.redhat.com/articles/2022/06/02/use-compil...

Unfortunately, C++ programmers are often extremely resistant to the idea that they ought to fix stuff like this. Even in a corporate environment where they are literally being paid to do what they're asked, there may be rebellion against the idea that the program ought to actually be correct (e.g. all tests complete successfully under UBSan) rather than just "Seems OK to me" as the standard. For a hobby project like OpenRCT2 it's often an insurmountable climb. Now, I don't know about that project in particular, but for a random Free Software project in C++ my opening assumption would be that they'd take a bug fix (like the one in the story) but they aren't likely to be sanitiser clean.

Not sure what industry you're coming from, but in games it's quite common nowadays to at least try to stay on top of asan warnings and purge them before big releases. It's a very easy way to prevent crashes and bad behaviour that might bite you unexpectedly in the next build when memory layout shifts just right to make the use-after-free someone else left in there break some important structure in unrelated code.

> Unfortunately, C++ programmers are often extremely resistant to the idea that they ought to fix stuff like this.

[citation needed]

People tend to take this stuff seriously if you're working with Good codebases. These generalizations are categorically incorrect, from my experience.

Putting emphasis on Good codebase. I'm afraid in practice good codebases are few and far between (the companies). Doesn't mean we should not try to make them better.

Reply on news.ycombinator.com