How Diablo Was Reverse-Engineered Without Source Code [video] | Hacker News Reader

How Diablo Was Reverse-Engineered Without Source Code [video]

youtube.com

403 pointsbane66 comments

I really appreciate the rise of YouTubers that are fairly knowledgable in retro gaming and computing niches. David Murray (of The 8-bit Guy) and Clint Basinger (of Lazy Game Reviews) are two other YouTubers who consistently put out interesting content about retro stuff. Of course they all have their areas of expertise and interest, and sometimes there’s things vague or occasionally outright incorrect, but I think the entertainment value more than makes up for it.

This video is really cool, though. It’s full of interesting anecdotes, and I enjoy the visual content showing things like looking at the binary in Ghirda.

Even though the answer does just boil down to discovering debug symbols and a version with debug assertions enabled, I still recommend watching as they go above and beyond, especially near the end when introducing some of their own work.

(Edit: fixed spelling error in Clint’s surname.)

Clint and his fascination for the 486-era machines is what made me assemble one similar to my first PC as a hobby, I love to look around and remember how webdev or even simple desktop apps worked in the days of old. Also it's Basinger with one s. ;)

Clint definitely rekindled my interest in running DOS/old Windows. There’s something hugely cathartic about going back to Visual C++ 6 on a Pentium, I have fond memories of being extremely confused constantly at all of C++ the language, MS’s compiler for it, and the Win32 APIs (MFC and ATL notwithstanding...)

I didn’t ever have a 486 or Amiga personally, but I look forward to trying some of these machines out as the opportunities arise. Even without the nostalgia and fond memories, I’ve found exploring older computers and software to be just as fascinating.

Also, thanks, fixed spelling. :)

A good way to experience the Amiga (and much cheaper than trying to get the relevant real hardware together) is to purchase and install a copy of the AmigaForever distro:

https://www.amigaforever.com/

Technically, you could probably assemble your own version from various sources on the internet, but if you just want to get up and running quickly, AmigaForever is one of the best ways to do it. If you drop it on a modern machine, you'll even have something very close to recent Amiga hardware releases (yes, there does exist modern Amiga hardware - everything from new motherboards to complete systems - but they aren't cheap to purchase).

I'll recommend GameHut[1], a channel run by Jon Burton[2] who worked on a lot of classic titles himself. The clickbaity titles and thumbnails are a little unfortunate, but there's a lot of interesting content on how things were programmed, with all the details broken down. e.g. 'Mickey Mania's "Impossible" 3D Chase'[3].

[1] https://www.youtube.com/channel/UCfVFSjHQ57zyxajhhRc7i0g/vid...

[2] https://en.wikipedia.org/wiki/Jon_Burton

[3] https://www.youtube.com/watch?v=nt-AxAqlrOo

> The clickbaity titles and thumbnails are a little unfortunate

They bring views which helps him make more videos.

One day enough people will see through clickbait titles that they won't work anymore. At least I hope so. It may be happening already, I don't know.

If that happens, it will be yet another example of optimization for the short term at the expense of the long term, and people still will fail to understand why that tradeoff is bad. Sure it's great for current-day you, but future you will pay, somehow.

How will the future "you" be paying? Not really clear what you mean. I don't see the short vs long tradeoff.

It will skew your evaluation of the importance amd veracity of information, so that future you will either not notice important things that aren't clickbaity enough, or if you resist clickbait, might not see important things that are too clickbaity.

But that's me as a viewer. For a channel I'm not seeing the long-term downside. If clickbait goes very out of style, they can just follow the trends, no?

There are many people who do not understand that optimization for the short term is paid for in the long term. Far too many. I don't understand how people do not understand that, and I recognize that, apparently, most people do not.

All decisions have (or are) trade-offs. Cause & effect. Since a cause cannot have an effect in the past, all causes have effects in the future, be it one nanosecond in the future or one thousand years in the future. That is to say, all decisions have consequences, and those consequences are always in the future.

If you optimize for the short term, you are, by definition, not optimizing for the long term. So, short term optimization necessitates long term de-optimization.

Examples:

Spend money now and have none for retirement (short term benefit), or save money now, and have it during retirement (long term benefit)?

Pollute the air now, earning maximum profit at the expense of air cleanliness in the future, or pay a lot of money today to find cleaner ways to operate, and have clean air in the future?

Fully invest in clickbait headlines today, earning ad impressions today, and risk clickbait fatigue later, or find a longer term strategy to retain readers in the long-term?

I can't readily think of an example decision that optimizes for short term gain that does not also optimize for long term loss.

My problem with this is that the short term is indeed very short, and the long term is very long. Short term optimization has long term consequences, and I would prefer short term consequences over long term consequences.

1. Without marketing hacks, like clickbait, there would be no business to optimise in the first place. Good luck optimising your non-viewed channel. You understand long term is only relevant if you survive the short term.

2. In the long term, you are dead. Optimise that.

>Since a cause cannot have an effect in the past, all causes have effects in the future, be it one nanosecond in the future or one thousand years in the future.

Off-topic, but you might enjoy reading about the delayed-choice quantum eraser experiment [1].

According to Wikipedia: "While delayed-choice experiments have confirmed the seeming ability of measurements made on photons in the present to alter events occurring in the past, this requires a non-standard view of quantum mechanics.", but it's still a mind-bending result.

1: https://en.m.wikipedia.org/wiki/Delayed-choice_quantum_erase...

> If you optimize for the short term, you are, by definition, not optimizing for the long term. So, short term optimization necessitates long term de-optimization.

I don't think " short term optimization necessitates long term de-optimization" follows. They can be both optimal.

> Spend money now and have none for retirement (short term benefit), or save money now, and have it during retirement (long term benefit)?

Spend money now on something that increases your chances to get more money (easy example education?) and benefit now and at retirement.

I've noticed that a lot of the links containing click bait titles actually list the hidden content in the URL. It's not all the time, bit frequent. Especially ones along of "You'll never believe what <celeb> did" and then the URL has the name and "married" or something. I never click on a clickbait URL without waiting for the full URL to expand on hovering.

“Clickbait” is all a spectrum though. The things we think are click bait today could completely pale in comparison to what we think is click bait tomorrow. And that spectrum really varies based on the audience.

PT Barnum has a quote for you...

LMGTFY... “Nobody ever lost a dollar by underestimating the taste of the American public.” ― P.T. Barnum

Unless this is a joke, I thought the famous PT Barnum-attributed quote was “There’s a sucker born every minute.”

My politically correct spin on that: "There's a customer born every minute."

http://www.thisdayinquotes.com/2011/09/no-one-ever-went-brok...

TL;DR: Not Barnum. Paraphrased from H.L. Mencken.

I suspect these things change mostly in generational waves. Children have a tendency to see things exactly for what they are.

As long as most endeavors are relatively short-lived, chasing near-term gains for long-term losses is the smart bet.

Yes, but they give me a sense that the videos will be similarly over-dramatic and dumbed down, and usually I skip past recommended videos that have them. In this case the actual content is well worth watching.

The 'unfortunate' part to me is that's it's good content presenting itself like bad content.

Retro Game Mechanics Explained is also very good. The creator is anonymous AFAIK but the quality is top-notch.

https://www.youtube.com/channel/UCwRqWnW5ZkVaP_lZF7caZ-g

Of course, nothing wrong with remaining anonymous or pseudonymous online. It comes down to many factors what an individual may feel comfortable revealing to a public audience. I do like to get to know YouTubers when possible just because a lot of them end up being pretty interesting, but it's not necessary.

The 8-bit Guy's work on Planet X3 (and the previous titles) is straight up amazing. I have such admiration for people like him.

https://www.youtube.com/watch?v=szhv6fwx7GY

I agree with the 8-bit guy but LGR is just plain wrong a lot of the times.

Let's not forget the best in the game. XboxAhoy aka Ahoy: https://www.youtube.com/channel/UCE1jXbVAGJQEORz9nZqb5bQ

Ross Scott (Accursed Farms) is one of my favourite (retro) gaming related youtubers, he always delivers incredibly well researched facts for every game in his "Game Dungeon" series.

https://www.youtube.com/watch?v=CLk64lmMtF0

I love Lazy Game Reviews. One of the most interesting channels on YT. I found him via a video he did on an old PC game series calls Petz, which still has some of the most eerily life-like animation of any game I've ever played.

Tl;DW: There were versions of the game released that still had debug symbols.

To be more particular:

- The Japanese Playstation port contained debug symbols

- A Windows debug binary with all assertions in tact was found

It is my understanding that this combination allowed for a better understanding than just symbols alone.

(Could be a little fuzzy on the details; I watched this video last night, not just now.)

To give further background, the Devilution team has primarily relied on these resources:

1. The Japanese Playstation port with debug symbols contained in `DIABPSX.SYM`. (see [1]).

Example debug info of the Cathedral dungeon generation algorithm:

  // address: 0x801259D0
  // line start: 612
  // line end:   624
  void DRLG_L1Floor__Fv() {
   // register: 19
   register int i;
   // register: 20
   register int j;
   // register: 3
   register long rv;
  }

2. The debug release of the PE executable, which contained assert strings (see [2]).

Example assert string:

  "plr[myplr].InvGrid[i] <= plr[myplr]._pNumInv"

3. The Rich header of the PE executable, which details the exact version of the original compilers and linkers used to build `Diablo.exe` (see [3,4]).

Example information recovered from the Rich header of `Diablo.exe`:

  Id  Build  Count  Name       Description
   0      0    155  Unknown    [---] Number of imported functions (old)
   1      0    229  Import0    [---] Number of imported functions
   6   1668      1  Cvtres500  [RES] VS97 (5.0) SP3 cvtres 5.00.1668
   2   7303     29  Linker510  [IMP] VS97 (5.0) SP3 link 5.10.7303
   3   7303      1  Cvtomf510        VS97 (5.0) SP3 cvtomf 5.10.7303
   4   8447      2  Linker600  [LNK] VC++ 6.0 SP3,SP4,SP5,SP6 link 6.00.8447
  48   9044     72  Utc12_2_C  [---] VC++ 6.0 SP5 Processor Pack
  19   9049     12  Linker512        Microsoft LINK 5.12.9049

4. Discovery of the original set of compiler flags used to build `Diablo.exe` (see [5]).

Primarily "/O1" was used, but there are also peculiarities such as the use of both Microsoft Visual Studio 6 and Microsoft Visual Code 5 for linking the game.

5. The heartfelt dedication of a team of people. GalaXyHaXz did the initial heavy lifting and succeeded in the tremendous task of getting the decompiled source code of Diablo 1 compiling with the original toolchain. Later on she released the project open source and a community of open source collaborators formed. Most of us have never met in real life prior to joining the project, which stands to show that there is strength in online collaboration that transcend both culture and borders.

6. The Beta release and the Alpha4 release of Diablo 1 has also proved invaluable resources for cross-validation as the compiler optimization level was not set to release mode for these binaries.

Interestingly, in the process a number of bugs in the original implementation of Diablo 1 were discovered. These have been documented in the source code of Devilution with `// BUGFIX: foo` comments, and have also been detailed in [6].

To track the progress of the project, the "Binary identical functions" milestone has been used in tandem with an assembly diffing tool developed in Rust (see [7,8]).

Anecdotally, it was an incredible moment when we first managed to run the cross-platform port of Diablo 1 (DevilutionX, see [9]) natively on Linux and succeeded in playing a multiplayer game connecting our computers in Korea and Denmark. It is equally thrilling to see the modding and porting community picking up the torch and already succeeding in porting Diablo 1 to Nintendo Switch!

The main reason for conducting this bit of software archeology is to preserve the classic title that is Diablo 1, for generations to come. And to revive it for modern hardware platforms and make it more mod-friendly in the age of open source software.

Happy coding! - The Devilution Team

P.S. the project README explicitly states that to play the game, you still need to have access to the original game assets released on the Diablo 1 CD. To acquire a legal copy, please refer to https://www.gog.com/game/diablo

P.P.S. for the verification process, there have been proposals that are both ambitious at a level of PhD research (see [10]) and that made us feel warm and fuzzy <3 In the end, many of the techniques outlined were discussed mostly on a design level, some were included as Proof of Concepts, but most of the work in reverse engineering Diablo 1 was from tender labour of a team that care for Diablo 1 the way you would your firstborn child.

[1]: https://github.com/diasurgical/scalpel/blob/master/psx/_dump...

[2]: http://diablo1.se/notes/debug.html

[3]: https://github.com/diasurgical/devilution/issues/111#issueco...

[4]: http://bytepointer.com/articles/the_microsoft_rich_header.ht...

[5]: https://github.com/diasurgical/devilution/issues/111

[6]: https://github.com/diasurgical/devilution/issues/64

[7]: https://github.com/diasurgical/devilution/milestone/3

[8]: https://github.com/diasurgical/devilution-comparer

[9]: https://github.com/diasurgical/devilutionX

[10]: https://github.com/diasurgical/devilution/issues/171

Wow! Thanks for this information, this is very cool. Software preservation is a big deal and it's very hard work, so huge kudos to everyone involved.

And DevolutionX even works on Haiku too.

Super noob question but i'm trying to wrap my head around how one could figure out the source code just based on things like debug info and assert strings? I watched the video and am staring at your examples and I just don't understand how you go from those to actual source code.

In the disassembly we can see a bunch of fine grained operations but the meaning behind them is opaque. For example, we see two array access operations but its not clear what they do. They might look like this:

mov al, [array + ebx]

Considering the assert statement from point 2: "plr[myplr].InvGrid[i] <= plr[myplr]._pNumInv"

From this we can see what the variables were named in the source code. Assuming "plr" = player and "InvGrid" = Inventory grid, we can deduce one the array access operations is to get the current player and another is for getting an item from the inventory grid.

Is there a version that runs on Windows 10? Its the game I used to initially addict my wife to gaming, and we'd like to play it again.

EDIT: Found it:

https://github.com/diasurgical/devilutionX/releases

Debug assertions in game binaries for blizzard and blizzard-branched studios are more common than you might expect. At least at ArenaNet, it was studio philosophy to ship with most debug assertions enabled so we could gather crash reports in the field to fix bugs. We only exempted assertions where the checks would cause performance issues (i.e. in a renderer or audio mixer). Most of our engineering culture was inherited from Blizzard and I think that particular rule was too.

Cool insight to hear. Apparently for the original Diablo, the usual release builds actually didn't contain debug assertions. I guess it must've become a part of their culture sometime after the release of the original Diablo?

I definitely believe you, though. I am pretty much positive that I've seen Warcraft III's engine crashing on debug assertions before when playing with the scenario editing tools.

I wouldn't be surprised if Diablo development is why the policy was in place. One of ANet's founders did a lot of work trying to help ship Diablo and it sounded like it had some serious engineering and quality culture issues (iirc Diablo was originally being developed by another studio that Blizzard acquired, but I might be misremembering)

I also remember Diablo II throwing up assertions when I was playing with the parameters in the txt files. Fun times.

The same thing happened for Oni. The debug symbols from a Mac version of the game were crossreferenced with dissassembly.

This is such a huge thing. If you are trying to reverse engineer something, spend time looking for other versions or test releases that might be out there. I've had a lot of success with this method.

I've been watching all of Modern Vintage Gamer's content for about 9 months now. This guy puts out some really nice videos with high production quality. There was a great video about game disk copy-protection a few weeks back. Awesome stuff, and very well researched.

Some other really good content out there is Game Historian, RetroRGB, My Life in Gaming, and Wrestling with Gaming. If you're into hardware restoration then Retro Man Cave has a lot of content too.

The DF Retro series on the Digital Foundry channel is great too.

And RetroAhoy on the Ahoy channel.

The original Mac version of Diablo also came with debug symbols, at the time I had written some sort of 'companion cheater map' thing that piggybacked onto the game for doing various nefarious things. It was called 'Dieblo' !

I also had to reverse engineer a whole lot of things, but at the time with Macsbug and a little bit of patience (and an OS with no memory protection whatsoever) it was pretty trivial work.

It's much easier to do reverse engineering on programs compiled with old compilers, nowdays compiler are really good at optimizing shit, which means making the assembly code more complex, using new instructions etc...

I love open-source mods/remakes of legacy games. It requires technical expertise and exceptional patience to build something so large in scale and complexity. The disgusting amount of work to accomplish this compounded by the fact that it's free for everyone makes me envious of their passion. A technical feat that is equally admirable as it is impressive.

Doesn't "reverse engineered" imply that source code wasn't used?

well, source code wasn't used was it?

Anyway, I think "reverse engineering" is a broad category that include disassembly techniques. "Clean room reverse engineering" is a stricter idea.

I'd call it a pleonasm [0]: "In particular, pleonasm sometimes serves the same function as rhetorical repetition—it can be used to reinforce an idea, contention, or question, rendering writing clearer and easier to understand."

[0] https://en.m.wikipedia.org/wiki/Pleonasm

I wonder why blizzard didn't just release diablo as open source.

I assume there's no strong minded folks at the top like John Carmack.

Id Software went through a little turmoil when he wanted to release the doom source, but it sort of highlights his unshakable confidence.

They may not have the original source. They lost quite a lot of the original Starcraft Assets and they were only later found by lick in someone's garage or loft well over a decade later IIRC.

I have a lot of code myself (mostly perl scripts, very old .NET and Java code) for old systems that are deployed that weren't in any sort of source control until recently.

This may surprise some but until 2006-2007 most source control systems were by today's standard terrible.

I've been through the torture of working sometimes with Accurev and Microsoft Sourcesafe and tbh you are better just zipping up your source code and putting a date and time on the archive name. Even more modern systems like Team Foundation Server are painful.

Semantic question but doesnt reverse engineered already means you have no access to source code?

Yeah. On the other hand probably most of us had a chance to work with code that needed some reverse engineering to be understood ;)

Typically yes. And they didn't have source code access here.

I'm pretty happy calling disassembly of a compiled binary reverse engineering. For sure it's arguably not "clean room" style, but not all reverse engineering need be.

So is Devilution considered the current best port by the Diablo community? When I last checked there were at least 3-4 others.

Ah, the good old mpq files... I remember extracting sound/music from Warcraft 3 and using them as ringtones.

Aha! So game developers do write tests after all :)

This sounds lees like reverse engineering and more like decompiling.

Decompiling (and then turning the resulting mess into something usable) can be a significant part of reverse engineering.

Yeah, this was hotly debated on Reddit that it wasn’t truly reverse engineering.

Well it is by most definitions tbh. Tbh the people on Reddit aren't the brightest.

Amazing video ;)

Reply on news.ycombinator.com