Reverse Engineering for Everyone
0xinfection.github.io
0xinfection.github.io
Andrew Tridgell explaining how he reverse engineered Microsoft's SMB protocol with the "French cafe technique": https://www.samba.org/ftp/tridge/misc/french_cafe.txt
Tridge also reverse engineered BitKeeper, the proprietary software that Linus foolishly used to host Linux kernel development for a while. He noticed that if you telnet to the BitKeeper address:port rather than use its proprietary client, you can type "help" and it then spits out a list of commands to try...
You can then interrogate the repository with these commands and get a complete understanding of all the internal data structures, without ever using the proprietary software, let alone having to disassemble it.
The fact that Tridge did this reverse-engineering led BitKeeper's owner, Larry McVoy, to rescind the Linux community's use of his software, so Linus wrote git.
I wouldn't necessarily call it 'foolish': Linus used the best available tool at the time. (I don't know whether BitKeeper was the best available tool in some absolute sense, but Linus looked around and evaluated many of them.)
> [...] you can type "help" and it then spits out a list of commands to try...
That was actually a nice engineering / UI decisions by the BitKeeper developers, but I'm afraid the morale of the story would be not to make your software too helpful?
> You can then interrogate the repository with these commands and get a complete understanding of all the internal data structures, without ever using the proprietary software, let alone having to disassemble it.
That's a strange use of 'use'? Clearly, talking to some software over the network is 'using' it?
The point is that the proprietary client software was not used.
It was foolish because, by selecting it, Linus was endorsing use of non-free software to work on one of the premier free software projects. It gave Larry McVoy unwarranted control over Linux kernel developers.
As rms said at the time: "The spirit of the Bitkeeper license is the spirit of the whip hand. It is the spirit that says, "You have no right to use Bitkeeper, only temporary privileges that we can revoke. Be grateful that we allow you to use Bitkeeper. Be grateful, and don't do anything we dislike, or we may revoke those privileges." https://marc.info/?l=linux-kernel&m=103454948625224&w=2
It caused animosity for years and was resolved by Linus writing git, famously in 10 days. Could he not have taken 10 days off in 2002 and written his preferred DVCS then?
The moral of the story is: don't use proprietary software. It will bite you in the ass.
"Torvalds seems to have fallen for the “free beer” argument: He didn’t have to pay for BitKeeper, so he figured it was good enough. But not having to pay is not, and has never been, the real purpose of free software. The point is to avoid the situation Torvalds eventually found himself in: McVoy didn’t like how his product was being used, so he took his ball and went home. Could you afford to switch gears in the middle of a project if one of your key software vendors did the same?" https://www.infoworld.com/article/2211030/linus-torvalds-bit...
> That's a strange use of 'use'?
To "use" software is to copy it into your computer's memory/CPU to execute it, for which courts have said you need a copyright license. You don't need a copyright license to connect to an open network port and interrogate it (or even capture packets of other people's conversations). US courts have also affirmed that web-scraping is a legal way to collect information because you're just throwing the data out there to anyone who asks; if you want to force people to agree to terms and conditions to see data or "use" web-software, you have to make them login or supply a key that you only issue _after_ they agree to your license or contract.
Andrew Tridgell did not even use anything which would require him to accede to Larry's license. It wrecked Larry's desire that nobody work on a "competing" tool to his, and there was nothing he could do about it, which is why he took his ball and went home.
"So I’d like to stress that while it really came together in just about ten days or so (at which point I did my first kernel commit using git), it wasn’t like it was some kind of mad dash of coding. The actual amount of that early code is actually fairly small, it all depended on getting the basic ideas right. And that I had been mulling over for a while before the whole project started. I’d seen the problems others had. I’d seen what I wanted to avoid doing." https://www.linuxfoundation.org/blog/blog/10-years-of-git-an...
In some sense, yes. But I wouldn't say my mom uses Linux when she uses her Ipad to visit a website hosted on a Linux server.
In this case, the software (or a component of the software's ecosystem) was "in use" over the network.
It goes to show that yes, you need someone to score, but you also need someone to make that critical pass of the ball.
Is git a "clone" of bitkeeper?
1. Learning how to use Cheat Engine to scan video game process memory and modify games.
2. Learned how to read/replay packets in an MMO to try an cheat.
3. Learned how to craft DLLs, hooks and inject them in processes.
4. Learned how create patches for executables to solve some crackme challenges.
5. Mess with real world software that requires a license key, to suddenly not require a license key (or accept any key).
6. Mess with binary formats to try an reverse how game saves worked to.. you guessed it, cheat.
7. Get a real job and make money with the skills and knowledge I acquired.
I think in reality it's the type of thing you do just have to try and spend some time on. The OP tutorial comes across as very sparse, both trying to cover too much and also not really teaching reverse engineering skills more than most people would be able to pick up in a few hours of messing around. beginners.re in contrast is massive, but also much more in-depth and goes step-by-step; on the other hand crackmes are probably better hands on challenges to try.
Disassembling a large binary would get you a massive text file that was painful to navigate - and often times I'd find that the code I was interested in removing "Invalid license key" (ahem) would be stored in some unrelated DLL.
So for me setting breakpoints on MessageBoxEx, and similar things, was by far the quickest and easiest way to go.
I have an ongoing reverse-engineering project for a video game and I ended up getting in contact with a self-taught modder of the game, who doesn't know how to program. He learned more in a couple of evening Discord calls with me showing him around the reverse-engineered Ghidra project, explaining the basics of computer program engineering as we went, than he did flipping bits with Cheat Engine.
He then proceeded to recreate a fairly ambitious mod that was showcased in a Youtube video 15 years ago but never released, something that was bugging him for years but was unable to recreate. I steered him throughout, but by seeing how the pieces fit together he then managed to do the same mod on the sequel (which was never done before) all by himself.
Experience with engineering gives you perspective when reverse-engineering.
I have met many software developers that have almost no understanding about that stuff. They wouldn't help much when it comes to reverse engineering.
At the end of the day, there's a bunch of knowledge you need to be able to reverse engineer efficiently. It doesn't really matter if you're coming from flipping bits in CE to programming or vice versa, but you need both. Having some around that knows both guiding you is a massive help.
For what it's worth, I also started reverse engineering first and programming second. There were many concepts I knew but didn't know the name of. I remember seeing a weird function where a pointer to an object was passed via ecx. I had no idea that how functions were called was a "calling convention" and that Microsoft called that a __thiscall. But at the end of the day, I did figure out what was going on, I just couldn't tell you what the original c++ code was until years later (when I finally "learned" c++).
That being said, I believe that there's a large skillset overlap between comparable reverse-engineering and programming activities. Knowing various programming patterns and architectures is helpful for making sense of (de)compiled code during static analysis. Being knee-deep in the bowels of a misbehaving program armed with GDB and you're getting a taste of dynamic analysis. Throw in some missing debugging symbols or advanced optimization work and you'll pick up some assembly on the way.
In my eyes, the only real difference is the mindset. On one side you're building software, on the other you're deconstructing it. Maybe I've been at it in the trenches for so long that I can't tell the difference anymore.
I think a fair point is that there are common idioms that you need to learn one way or another. Whether that is formal training or intuition or just plain force of will, you need to come to understand the meaning of what you are looking at and not just what the individual instructions are doing.
Otherwise, it's a similar idea to saying, "nobody needs to learn how to read music because look how great Jimi Hendrix was and he couldn't".
I'm not suggesting that aspiring reverse-engineers need to pursue a full-blown CS degree first, but most reverse-engineering activities usually have at least one counterpart engineering activity. You can power through without learning it first, but I'm not convinced that it's easier or faster to learn that way.
As for me, I've spent quite a lot of time doing low-level software engineering beforehand (stuff like OSDev, bare-metal programming and GDB debugging sessions with missing symbols...) and I've picked up on reverse-engineering very quickly I believe, thanks to lots of relevant prior engineering experience. Had I spent my time making cracks and mods instead, I highly doubt I would've been able to later pivot towards software engineering that easily, due to a lack of foundational CS knowledge.
I never suggested people learn entirely on their own. I learned in a loosely structured way by reading thousands of forum posts, asking questions on forums, sitting in IRC channels talking to people, etc.
everything you listed here could be in a book used to help you gain those skills
Although some books greatly help in getting there, I learnt a lot from "Reversing: Secrets of Reverse Engineering" by Eldad Eilam (might be a bit dated now)
Do you mind sharing what kind of job is that?
It was a chaotic mess of C++.
I did it for a year before joining a startup, and on and on.
Much better resources are Eldad Eilam's "Secrets of Reverse Engineering", for Windows "Practical Reverse Engineering" and for the absolute basics, Patterson's "Computer Organization and Design".
It also linked to this resource, which was more in depth... https://github.com/mytechnotalent/Reverse-Engineering
EDIT: Whoops... it looks like it mostly links back to the original article.
It's in web archive now, https://web.archive.org/web/20191201105759/http://search.lor...
Very long, windy and hard to parse sentences.
For example, Part 2
> There are two basic techniques that you can employ when analyzing malware. The first being static analysis and the other being dynamic analysis.
> Static analysis uses software tools to examine the executable without running the actual decompiled instructions in Assembly. We will not focus on this type of analysis here as we are going to focus on actual disassembled binaries instead however in future courses we will.
> Dynamic analysis uses disassemblers and debuggers to analyze malware binaries while actually running them. The most popular tool in the market today is called IDA which is a multi-platform, multi-processor disassembler and debugger. There are other disassembler/debugger tools as well on the market today such as Hopper Disassembler, OllyDbg and many more.
> A disassembler will convert an executable binary written in Assembly, C, C++, etc into Assembly Language instructions that you can debug and manipulate.
> Reverse engineering is much more than just malware analysis. At the end of our series, our capstone tutorial will utilize IDA as we will create a real-world scenario where you will be tasked by the CEO of ABC Biochemicals to secretly try to ethically hack his companies software that controls a bullet-proof door in a very sensitive Bio-Chemical lab in order to test how well the software works against real threats. The project will be very basic however it will ultimately showcase the power of Assembly Language and how one can use it to reverse engineer and ultimately provide solutions on how to better design the code to make it safer.
> In our next lesson we will discuss various types of malware.
could be written:
> There are two basic techniques that you can employ when analyzing malware: static analysis and dynamic analysis.
> Static analysis examines the executable without running it. We will not focus on this type of analysis here, however in future courses we will.
> Dynamic analysis uses disassemblers and debuggers to analyze malware binaries while running them.
> A disassembler converts an executable binary into Assembly Language instructions that you can debug and manipulate. There are many disassembler/debugger tools available such as Hopper, OllyDbg, IDA and many more. The most popular being IDA, a multi-platform, multi-processor disassembler and debugger.
> Reverse engineering is much more than just malware analysis.
> At the end of our series, we will use IDA in a fictional scenario where you will be tasked by the CEO of ABC Biochemicals – a very sensitive Bio-Chemical lab – to ethically hack his company’s bullet-proof door control-system.
> The project, while basic, will showcase the power of Assembly Language and how one can use it to reverse engineer black-box binaries and ultimately find solutions to make the code safer.
> In our next lesson we will discuss various types of malware.
That's because you're supposed to reverse engineer them :)
Reverse engineering in the real world takes a few forms, some of which the write takes on too briefly towards the end of the material. Applied reverse engineer is usually modifying an existing piece of software so:
* .dll/lib injection
* signature scanning and patching
* packet interception and rewriting
* mitm HTTP(s) calls
I'm curious if there's any reading out there that covers this stuff from the meat and potatoes and less of this CS 101 stuff.
I've done all of the above, and you can usually learn about this stuff from some different forums on the web, but I don't know of any good bibles on the subject matter.
Honestly, I think the most effective way to learn about how to reverse-engineer something is to learn engineering at the same layer first and then start tinkering. If you want to binary patch a program, learn assembly. If you want to inject a .dll, learn how to write and use dynamic libraries. If you want to MITM a REST API, learn how to call a REST API. Because once you know the rules well, you can start breaking them and see exactly how much you can get away with.
I wrote a series of articles on reverse-engineering on my blog, about studying and modifying a program that outputs an ASCII table, mostly because I needed a way to introduce delinking as a technique. I would not say it's good, but it starts with how to build the case study and then it handholds the reader through the meat and potatoes.
A binary is likely to have a reasonable amount of often-called code for memory operations (memset, memcpy, strcat, strlen, sscanf, log) and a lot of library code (Flexcomm_Init, Clock_AttachClk, SPI1_Handler, NVIC_EnableIRQ) and then probably fairly little actual application code. For Ghidra users, being able to ignore the boilerplate (mem and BSP code) and quickly find and analyze the application code saves a TON of time.
(Conversely, if I know a binary is written using FreeRTOS, finding the task creation function would be my first step, as this reveals nearly all of the application code.)
There are techniques to help (setting a flash memory region as non-write so string references are recognized and disassembled correctly, loading a chip SVD so all the library code is more obvious) but those come with experience or a good hands-on tutorial, and they still won't tell you everything about the application code.
In my own breakdown of one Cortex-M binary (bare metal, no objects known) the only reason I was able to get the firmware in the first place was by noticing and decoding a base64 string in an unpacked Electron app used for USB communication with the device. This ended up holding plaintext credentials for their update server which had two channels: one for encrypted production binaries and the other for unencrypted development binaries.
In this specific case, it helped to know what base64 looks like, but that's like how knowing different methods of slicing onions might help you figure out a recipe by tasting a cooked meal. Very often such background knowledge is irrelevant. Once in a while it will be the only realistic way forward.
Would you mind sharing the links? I would be interested!
I expect that you'll be mostly interested in parts 2 through 6. Part 1 explains how a toolchain works in general (so mostly CS 101 stuff as the OP put it). Parts 7 to 10 demonstrates the delinking technique by easing into it, a technique which is as powerful as it is esoteric, but probably not what you're looking for in a beginner's guide.
In my experience using radare2 to peek at the code is pretty much the meat and potatoes of reverse engineering binaries and far from "CS 101 stuff". You certainly don't need to modify a binary to MITM an API or inspect/alter packets or inject code via dynamic loading; nor is it the most convenient or clean or easy to maintain way to do so.
Secondly, this is a shockingly dismissive attitude for such a large resource. It took me a few minutes to just read through the table of contents.
I mean, you said you read the table of contents, yeah? Doing the same thing across different CPU architectures isn't doing something at length, it's just doing the same thing over and over again in rhymes.
In practice, yeah, people in the wild are absolutely modifying binaries, injecting, stubbing .dlls and redirecting calls, or creating proxy servers that alter payloads, for sure.
Learning how to compile a program isn't exactly reverse engineering worthy content to write about.
Looking at a binary is like 99% of the work, though. Or at least looking at some secondary form of it (e.g. assembly, decompilation, etc). Tools are absolutely critical to the work.
> people in the wild are absolutely modifying binaries, injecting, stubbing .dlls and redirecting calls, or creating proxy servers that alter payloads, for sure
I would call modifying a binary "cracking" it but it's been a few decades since I was involved in that scene. I also think that the topic is large enough to warrant multiple focuses—to me, at least, writing a MITM server is much more trivial than extracting a private key from a binary (or a running process) that makes that MITM server functionally useful.
> Learning how to compile a program isn't exactly reverse engineering worthy content to write about.
That's a disingenuous characterization of most of the content here. Coding at the instruction level requires a different way of reading and writing code than you're otherwise exposed to. Most programmers aren't used to handling bits directly, and certainly not to the extent that it rewards you at the instruction level for learning and knowing. With the tools here you can, in fact, sit down and inspect the license verification function of a piece of software (although I'm not sure how much that's true or beneficial these days with code-signing etc).
EDIT: Or you could do what I did and work with as, `otool`, and a hex editor, and learn extremely slowly & painfully why custom-built reverse engineering tools are so valuable to learn.
There's always more to learn, of course, but that's no reason to belittle what you've already learned and other people still have yet to learn.
If you're going to target script kiddies, at least show them how to Hello, World! from a DLL_PROCESS_ATTACH, and then teach them sigscanning.
One can't necessarily build an airplane after watching a documentary on it.
Even if there was some "bible" on it, reverse engineering is one of those things that you have to put the reps in for to get good at it and actually develop understanding.
The "bible" is tackling reverse-engineering related projects independently over the course of months/years and picking up knowledge along the way.
Starting with something like cracking software (and making increasingly-advanced cracks) is always my advice for beginners.
funnily enough I have a team reverse engineering binary data formats, which is often more easily accomplished by other means + only dropping down to the disassembly/decompilation where absolutely necessary. and which as far as I am aware never involves binary patching
but yeah about the article it seems like if you know this much about assembly / chips etc. to be able to read it, then general problem solving ability should be able to cover most of the article's content
Ahh yes, the famous 8064 computer.
Just kidding, looks like a great work.