I used to work on Wine, first as a volunteer and later as a job, so spent a lot of time staring at gigantic multi-gigabyte sized logs trying to work out why an app was crashing when running on Linux. Sometimes apps would work fine for me but be reported as crashing by a user, or we wouldn't have access to the app at all, so logs were the only way to work out what was going wrong.

We got a bug report that an app would crash, and I couldn't reproduce it. So we asked the user, are you using the latest version of Wine from our website? "Yes I am". OK, that's odd, send us some logs then. The crash was some sort of memory corruption during startup of the app. Everything seemed to be running fine, the app was loading files and reading registry entries happily, and then suddenly it would segfault in a random place. No opportunity to debug directly, as everything was binary only and only crashing on this guy's machine.

I spent days working painstakingly through hundreds of millions of lines of API call traces, until eventually I found what seemed to be a difference between his logs and mine. In his logs, some registry reads were failing, and in mine they worked. But why?

It turned out that the guy had been lying to us. He hadn't actually installed the app using the Wine downloads from winehq.org, he'd installed it from the Debian repositories. The packages provided by Debian were badly broken: they had split various tools out into a separate -utils package which wasn't installed by default because that complied with Debian standards better. But that was an error because Windows doesn't care about Debian standards and those tools aren't optional there, so many programs assumed those tools were always available. One of them was regedit.exe, which this app's installer was running with some flags to add default registry entries. On Windows this would never fail, so the installer didn't check the error codes and the install failure was silent. And then the app didn't check the error codes when reading the entries either, because again, that would never fail on Windows. So the reads silently did nothing, the memory the app expected to be initialized wasn't, it tried to use it and corrupted its heap which then led to a random crash about a million API calls away. The original failure wasn't even in the logs I was looking at.

At the time we had an explicit policy of not supporting anyone who installed Wine from their distribution packages, exactly because of bugs like this. Instead the project provided its own apt repositories. The distro-centric model Linux used was just broken because it led to packagers who weren't a part of the upstream communities "fixing" software they didn't understand as they packaged it. The notorious SSH bug was another case of that but such stories are commonplace. Debian users in particular were hard to deal with because lots of community built packages was a part of the distro's appeal and moat, even though upstream developers often hated it (lots of obsolete bug reports or distro-created bugs). So they had become defensive, and some had taken to deceiving upstreams when filing bugs because they thought they knew better.

Needless to say, a multi-day memory corruption debugging session that ended with "there is no bug, follow the install instructions on our website and stop lying to us about it" was by far the most annoying bug I ever had to work on.