I think answering the questions about decryption and finding the key would make a pretty nice blog post, if OP has a blog.

Edit: Looks like he made a really detailed follow up comment describing the process. Impressive.

I wanted to dump sounds from the game, not for myself, but because I wanted a way for people to do videos/soundbanks without having to record them from the sytem audio, which would often contain other noises from the game. So that was my initial motivation.

I started by doing it the brute force way, load game, dump content of ram into a file, use a command line tool (can't recall name) that would search for media files inside that file (using magic numbers and whatnot), and boom. I would get a bunch of .wav files from the game. This worked, but now I had to categorize each file one by one, since it's just a data dump all filename information was gone.

Using an hex editor I found that the file names were also loaded into memory (ctrl+f .wav in an hex editor proved as much). So I needed to figure which file names pointed where, which was not an easy task looking just at hex values. I figured: well, the game has to know somehow which names belong to data, at least at some point, so I googled x86 reverse engineering and landed on a wiki page that brought me to a program I used only once a long time ago, ollydbg. Back when I was 14, (I'm 30 atm) I had followed some tutorial on how to use ollydbg to crack tony hawks pro skater 3. I wasn't too difficult, but I never followed up because back in the day I cared more about playing video games than figuring out how they worked.

Ollydbg this time proved fruitless to me, I had no idea what I was looking at. I followed some other tutorials and videos on youtube, the most helpful were the tutorials on cheat engine but I was still stumped on how to reverse video games, so I dropped the project, for a while.

Back then I had been recently introduced to python, which rocks btw, and one of the books, possibly the turning point of my reverse engineering "career" was a book called gray hat python. The book explained how debuggers worked, what's the stack and what does it do, registers and what they're supposed to hold, how breakpoints work, how hooking dlls works, etc. This proved the cornerstone for me understanding what's going on.

I've come to realize I haven't explained what ollydbg is. Ollydbg is a windows debugger used mostly by reverse engineers, with most-likely evil intents, to read machine code. A debugger is a program that tells the operating system "hey I'm a special program that wants to find bugs in this program so let me peek around", so it has access to what memory the program has allocated, or what files it has loaded, etc. It also converts the executable part of memory into assembly instructions. So instead of you seeing "57" you see "PUSH EDI". Basically, let's you read the assembly code of a program.

So now with my new found knowledge, I found ollydbg to be very helpful. I first hunted down where the game doing the file loading, which was easy because it's using windows APIs, then poked around until I found the call that returned the decrypted data, which was also easy. Knowing this information, I built a python script that would pretend to be a debugger, hook itself into the game, set a breakpoint right after the function call, retrieve all metadata and decrypted file data from registers/stack, and then resume execution so the next file could be loaded and repeat the steps all over.

So, mission accomplished right? Nope. A major pain in the ass still remained, this script would only dump files that the game would load, if the game didn't load the files, they would never be decrypted, this meant I had to select every "hero" and load every map, select every announcer, etc. Which I did for a while, but I figured, "there must be a way to automate this". So I tried to understand what the file loader was doing. This is where I spent the majority of my time, just poking around at code, tracing segment of code and reading the traces to try and figure out what was going on, etc. One key piece of information, the game was using the AESENC instruction to decrypt stuff, so I read some wikipedia articles on AES, got the basic grasp on how it worked, and tried to figure out what the game was doing. Note one thing, AESENC is used to encrypt files, however the game was doing decryption. This stumped me for a while. I watched free lectures on encryption online, I read articles, stackoverflow, youtube videos. None of them could explain why would you do encryption to decrypt a file. So I figured they must be using some sort of custom encryption, because according to wikipedia, one of the paremeters for the XOR right before the AESENC instruction should be the key, and it didn't work when I inputed it in online decryptors and whatnot, trying all combinations of operation modes, etc. I was stumped, and for several months I could not figure out how the game was doing it.

The next eureka moment was hiding in plain sight, a webpage that I had seen several times already, but never noticed a detail. It's even bolded. https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation... . In CFB mode of operation, the encryption algorithm is used to decrypt the file. A light turned on in my head and if what I thought was right i would have solved part of the problem, so using the graph on that page I quickly built a proof of concept to test my theory. And I was right. To explain what my theory was and why it worked, I need to explain a bit on how AES works. I'm strictly talking about AES-128 here to keep things simple. This comment is too long already.

So AES-128 can be divided into two parts, first part is the key expansion. It grabs the encryption key, and uses a special algorithm to transform it into 10 different keys. The second part is the encryption operation proper, it does 10 rounds each using one of the keys in the first step, first and last round are different, but the others you're just grabbing the result of the previous round and the round key, and doing an AESENC with those as parameters, the last round will output the decrypted data.

So my theory was, they don't touch the encryption proper, but alter the key expansion step. It I was right, I could grab the first block of the file, use the round keys and IV I got from ollydbg, and be able to decrypt a whole file. And like I said, I was right :) Next step: find out how the game is doing the key expansion.

This was kind of a boring job. I worked backwards from the encryption operation. See where the output was being generated, then replicate it in python. Remember, this is all in assembly, and optimized by a compiler, so the code is anything but structured and logical. After implementing a bunch of the code, it started to look familiar, it basically was a modified version of the regular AES key expansion. It seemed to skip some steps. The key that was fed to that key expansion, was being generated by running a sha-1 hash on the relative filepath of the file encryped. You can find these two functions here: https://github.com/Nodja/AwesomenautsFileDumper/blob/master/...

Apologies for the lack of comments and ugly code. It went through dozens of iterations and it's based on assembly code that I don't fully understand yet. Well I know what it does, but don't know why it was done that way.

From here on was more busywork, basically make the script read a whole game folder and dump the contents somewhere else.

TL;DR: I didn't need to "use" assembly, I didn't write any assembly, I however needed to understand assembly since I was reading decompiled machine code. This allowed me to understand what the game was doing when loading files which lead me to the encryption "key". It was not actually a key that I needed to find, since they were using a modification of the whole AES-128 algorithm that only needed a file name as input, instead of key and IV.

P.S. There were some white lies to shorten the whole thing. Basically just me coming to conclusions sooner that I did, and skipping some steps (the game used compression and their own archive format). Also apologies if I made some typing errors, I didn't proofread my comment and sometimes I type words that are different than the word in my head :P