Hacker News Reader

Comment by jtchang | Hacker News Reader

How would a program detect the use of a debugger? I know a lot of crackmes and other anti-piracy measures involve detecting the use of one but am not sure how they do it. Do they just look for running processes with a debugger signature like softice?

View on HN

One technique I remember being used in DOS apps from many years ago was that the code would be encrypted in such a way that the next instruction to be executed would only be decrypted immediately before it was run. This was achieved by setting up the single step interrupt as the decrypter, and running the code in single step mode.

The fact that the code was encrypted meant the debugger couldn't disassemble it in any meaningful way, and also made it impossible to set a breakpoint (since the breakpoint would just end up being "decrypted" into some other opcode that would inevitably crash). The debugger also couldn't step through the code, because taking over the single step interrupt would prevent the decrypter from running, so you'd just be stepping through garbage.

The way I worked around this was by writing a debugger that could hook the single step interrupt in such a way that it still forwarded the interrupt onto the previous hook. I still couldn't set breakpoints, but I could step through the code, watching it decode itself as it proceeded.

If you're single-stepping you don't need to patch the instruction stream for a breakpoint; you can just test the instruction pointer against the break address at each step.

Not every processor has the means to perform direct comparisons against the program counter; as far as I know 6502 and MC68000 families can not do direct comparisons with the program counter.

It's the program counter of the state that was interrupted by the single-step trap that's of interest, and that program counter is generally pushed onto the stack by the trap.

For example, this is how the trace exception on the M68k works - the program counter of the next instruction to be executed can be read off the stack by the exception handler. The 6502 doesn't have built-in software single-stepping but the same effect was sometimes achieved by tying a short timer to the NMI - and when the NMI is asserted, the interrupted program counter is pushed onto the stack.

Lots of different approaches: To figure out if a process is ptraced the process can try to ptrace itself. An executable can only get ptraced once at a time, so the second call to ptrace will fail. Of course you can workaround that check by patching the code so that it doesn't call ptrace.

Breakpoints are also easy to spot. Just look for 0xCC in the memory. But of course there are also multiple ways to work around those checks (such as using harder to detect hardware breakpoints instead of software breakpoints).

What I recall from the old days: it was indeed possible to detect Softice, through various ways, for instance simply trying to use its own API (through int3 as well having registers set to special magic values).

Another trick to detect breakpoints as an anti-debugging measure was to compute a key over the block of memory that was to be protected. Any int3s in there would give a different output. A countermeasure was to use "memory breakpoints" which work via the x86 debug registers: https://en.wikipedia.org/wiki/X86_debug_register

There are many different techniques that you can find in this paper (wich, I think, is a reference on the subject):

http://pferrie.host22.com/papers/antidebug.pdf

For a fresh compilation (which includes the previous paper) you can check this article:

http://antukh.com/blog/2015/01/19/malware-techniques-cheat-s...

> How would a program detect the use of a debugger?

IsDebuggerPresent() is the most straightforward way: https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...

The most straightforward way on one particular platform. There are definitely apps using anti-debugger techniques even on iOS.

An ideal debugger can't be detected.

In the specific case of x86 and int3 bps, you can abuse variable-length instruction encoding to jump into the middle of another instruction. Then code execution can differ depending on the presence of 0xcc.

One common trick was to make the code itself part of a calculation for decrypting/checksumming/encryption/etc., so any modifications would cause a silent failure much later.

The game King's Keep on Sinclair Spectrum was (in a way) like that. It had one or two obfuscated subroutines for xor encoding the main program. They were pretty well localized so once I identified them, decompiling the rest was trivial.

Fun times then and now :)

The most basic one asks the kernel if there's a debugger attached. Others check if certain programs are running. There's more complex ones, but I can't remember them right now

sometimes you don't have to bother detecting whether your program is being debugged, you can just adopt behavior that would make it impossible to run your code under a debugger.

i got my start in programming by trying to break various copy protection schemes in MS-DOS games. i used a software debugger that would set int 3 breakpoints, as the article mentions.

BUT, for the debugger to work, it has to rewrite the interrupt vector table so that int 3 instructions will cause the debugger's own code to be executed. really wily code would use the entry for int 3 in the interrupt vector table as a scratch variable to perform a bunch of its own calculations, perhaps thousands of them so you can't find and modify them all, which would ultimately wind up with an address to jump to, to continue execution of the program. this means that, when the debugger rewrites the int 3 vector, the program's internal calculations would be thwarted, effectively stopping the program at that point.

this was really difficult to get past. there were other people doing this kind of thing who seemed to find ways around this technique, but i never did.

Each process can be ptraced by only once process at any time, so if the malicious program want to avoid debuggers, they can attempt a ptrace() on their own process. If it fails, they know they are being debugged.

There's a bunch of ways. Some are obvious (like a call to IsDebuggerPresent()) and some not so much (like tripping some exception that only gets thrown when a process is not being debugged). Youtuber MalwareAnalysisForHedgehogs[0] has some interesting videos about this and anti-reversing tricks in general.

https://www.youtube.com/channel/UCVFXrUwuWxNlm6UNZtBLJ-A

It's essentially an arms-race. For every anti-debugging or obfuscation technique there's a reverse engineer who is willing to do hard work to overcome it. It becomes a kind of psychological warfare-- trying to wear-down the reverse enginner to make the give up.

If this kind of thing interests you check out 4AM's Apple II cracks. He's removing the copy protection on old software so the techniques aren't directly applicable to current machines, but there's a font of creativity in the code he's reverse engineering. https://twitter.com/a2_4am

Reply on news.ycombinator.com