Sorry, Wrong Number: Debugging a Crash Under Wine

Similarly, instead of doing "a few hundred stepis" to get to the call insn that jumps to a garbage address, you can let execution run to the crash point and the reverse stepi once. The "reverse" functionality parts of rr are one of its strongest features and well worth the time investment to get acquainted with them.

(I bet my own debugging workflows and habits are stuffed full of this kind of "works but is definitely not optimal" move, though, so I don't mean this as a criticism of the article.)

This is a good point, but I feel like there was a reason that I couldn’t do this. With Wine under rr, the replays were a bit tricky to get working in the first place. I think it is possible i could’ve done this still, but I first needed to find the number of events to skip for that part of the replay. See this issue:

https://github.com/rr-debugger/rr/issues/1687

By the time I got to the point where a reverse continue would’ve been most useful, I’d already left rr due to this issue.

I would start by running the game from the terminal and read all warnings and errors, compare those with games that work because some of the warnings are always there but don't affect the games.

I used to work on Wine, at first as a hobby and then later as a job (for CodeWeavers). As part of that I contributed to the Wine Developers Guide. The Wine docs to help new devs are (or were) remarkably good given the enormous complexity of the project. Unfortunately "how to fix a bug in Wine" can still devolve very rapidly into:

Step 1. Learn Windows programming.

Step 2. Then learn Linux programming, the Wine architecture, X86 assembly, low level debugging etc.

Step 3. Draw the rest of the owl.

Nonetheless it's a good place to start:

https://wiki.winehq.org/Developer_Hints

https://wiki.winehq.org/Wine_Developer%27s_Guide

The days when fixing games involved just straight implementation of simple library calls were mostly over, even back when I was doing this a long time ago. Broken apps or games are much more likely to be hitting obscure "bugs" in Wine (which can mean failure to replicate Windows bugs), or other even more subtle compatibility issues, like the one this blog post documents.

I once had a game that always crashed under wine. Instead of trying to understand the bug, I got as far as figuring out that this was a virtual function call into nowhere and then just 909090'd the crap out of it. The game then seemed to work perfectly and I couldn't even tell what was missing/different after removing a totally random function call.

I wrote an intro to Wine dev: https://www.codeweavers.com/blog/aeikum/2019/1/3/working-on-... It doesn't go into a lot of debugging detail, as that's its own huge topic, but may get you started.

If you're debugging a thing in Proton, you'll want to use that project environment instead of Wine's directly, but the concepts are the same.

That’s the weird part: MinGW does use DWARF, embedded into PE similar to how it is embedded into ELF, with just one little non-standard trick for long section names. On Windows, GDB supports loading these symbols just fine. But if I forcibly get them to load on Wine under GDB, it doesn’t seem to work even if the load seems to succeed. Confusing.

Edit: For "GCC" I meant of course "GDB"

I have a patch that teaches GDB how to do this here: https://github.com/JuliaComputing/gdb-solib-wine. Unfortunately, I don't really have the time to upstream this, but perhaps it's useful or maybe somebody sees this and wants to take on the upstreaming.

Ahh, sorry I never completed the story. The program was simply fixed. I believe the way they wound up fixing it was simply static linking libpng and zlib. I found another workaround too, which was a very obscure linker flag, but I can’t recall what it was as it has been a few months now. If you run into this problem with a random binary and can’t get it fixed properly, you can use Microsoft’s REBASE.EXE or EDITBIN.EXE to rebase the preferred base address to be within the range of a RIP relative CALL. If you are savvy with PE, writing such a tool by hand is very straightforward; you need to write a PE parser that can map virtual addresses, relocate as if you are a real PE loader, update the preferred base address, and write back to disk. The only case where something like this won’t work is when the executable has some kind of on-disk CRC checking, or something along those lines. (So probably, VMProtect and Themida encumbered binaries can’t be modified this way. But that’s OK, as most tools of this sort only works with Microsoft’s toolchain, which does not have pseudo-relocs.)

Actually, I do have an answer for that: debuggers don’t work very well under Wine. It would be nice to get them working better.

The standard tool to use for Wine is winedbg, but as I wrote, this doesn’t work here because there’s no way to break before the application entrypoint (which is too late.)