Debug your programs like they're closed source | Hacker News Reader

Debug your programs like they're closed source

jvns.ca

126 pointszimbu66830 comments

I used to work for a (no defunct) company, ARDI, that made a Mac emulator. We dynamically translated 68K to x86 and did a clean room reimplementation of Mac OS 9. (Yes, crazy.) Much of the job involved debugging the OS when running 3rd party software (binaries). The first step was to collect a syscall trace. Then lots of thought experiments. It was a great way to develop extreme debugging skills.

> clean room reimplementation of Mac OS 9

Wow. Was the project successful? Where can we read more about it? Was it a perfect implementation - all the images in the right places pixel by pixel?

http://en.m.wikipedia.org/wiki/Executor_(software)

Source is at https://github.com/ctm/executor

I think the OP is of by two OS versions, but I didn't work at ARDI, so I may be wrong.

No, it looks like you're right, I misremembered. I haven't thought about this in a long time.

Whoa, Cliff open-sourced Executor. I didn't know that.

> Was the project successful?

As a business? No. As a hack? It was compatible with quite a bit of software and with syn68k on a Pentium whatever of the time, it was faster than any Mac.

Your my hero, I used this to play Escape Velocity before the Windows port. Anyways, Mac OS 9 doesn't run on 68k and exucator didn't run PPC applications?

Haha, awesome! Glad you enjoyed it. Yeah, we did a lot of "testing" on games. Yeah, I misremembered, it was OS 7. As far as I know, there was no PPC version.

Syscall tracing works well for debugging applications or pieces of one that interact externally, or for ensuring that the OS is doing what you think it should, but I've found that more often than not the buggy part is "inside"; e.g. if an output file is incorrect, you would be able to see the incorrect data being written to it via the OS (which would've been clear from inspecting the file itself), but not what lead to that.

Coincidentally, I prefer to use an Asm-level debugger (WinDbg, OllyDbg) whenever I need to use one - which for debugging something I wrote, is not very often. Maybe it has to do with the fact that I started with Asm and disassembling other programs, so it's relatively straightforward to map between source code and the instructions the compiler generated. This also helped with figuring out a very subtle compiler bug - I know that they're quite rare, but when they do occur, they tend to be extremely difficult to figure out otherwise.

OllyDbg is pretty good but it doesn't work with 64bit applications. Can anyone point me to one that supports 64bit?

Here is a discussion from 6 months ago: https://news.ycombinator.com/item?id=8092273

> WinDbg, OllyDbg

As someone who seldom leaves higher-abstraction languages, I've had some fun using Cheat Engine against old single-player games. I remember being pretty proud of the first time I managed to replace a few instructions with a larger set sitting in a code-cave. (Nothing impressive for those who learned assembly in a structured way, of course.)

Ah, Cheat Engine. I used to use it for multiplayer flash or shockwave games, as well as a couple other games along the years. Poor Nitto 1320, we killed that game. Got me into disassembly and hacking at that level, which was a lot of fun too.

That's actually how the auto assembler in CE works. It replaces the selected instruction with a jump to a code cave and places a jump back to the original code at the end.

Great article, very interesting! I never knew about this :)

For those on OSX, `dtruss` seems to work in more-or-less the same way.

    sudo dtruss -t open ssh whatever.com

More info on stack overflow: http://stackoverflow.com/questions/1925978/equivalent-of-str...

On Mac OS X you really want to learn about DTrace.

dtruss is a DTrace script. See http://dtrace.org/blogs/brendan/2011/10/10/top-10-dtrace-scr... for more useful DTrace scripts.

This just changed my life, thanks so much

Also on OS X, `fs_usage` will tell you which processes are accessing which files. Very useful!

From what I understand it report all interactions with the filesystem after its launch, it does tell you what process is currently accessing which file. lsof does a pretty good job for that.

For those on Windows, the excellent System Internals suite contains "Process Monitor", which does pretty much what you'd expect, allowing you to see all sorts of file (and registry) activity.

Agreed. I get by without having things I'm used to on my linux system like strace, lsof and so forth with it. It has a pane that displays pretty much everything with a handle from non-IP IPC sockets, normal sockets, files, registry, threads, DLLs and even obscure stuff like event handles and security context tokens.

Process Hacker is also nice, and the last time I used it, it displayed even more information. Plus it lets you inject DLLs, scan memory, and stuff!

SysInternals procmon.exe and procexp.exe are the only things I really miss when on Linux (either developing or just troubleshooting Firefox being slow again.)

A few procmon features you probably weren't aware of:

- It can log your boot process (by loading a driver very early in the boot process)

- It stores a lot of details about every line of output, including a stack trace (!), accessible by doubleclicking

I miss Search Everything on linux. Nothing beats real-time indexing of the filesystem without overhead.

You could setup something almost like it. mlocate is the defacto indexed filename search for linux, and the update script skips directories that haven't been updated - so an update call is quite cheap. You could run it every 15m, and have 'almost' uptodate indexed search.

Most systems should have it setup to run daily out of the box.

There are also more sophisticated desktop search tools like Recoll, of course.

Got to System Internals for procexp and tcpview but I'll never forget the first time I ran procmon.

ps: Windows really isn't sleepy, a lot happen all the time it's a bit scary. I don't know how other OSes are behaving when idleing though.

Ah, an article on tracing.

I am very impressed by lttng myself:

http://lttng.org/

Everytime I read about strace I feel guilty about not having tried it out.

Another awesome story: http://lwn.net/1999/0121/a/mec.html

Even though I havent followed the above idea ( I will try to do in sometime ), I have followed this way, which have helped me a lot.

I used to make guess when I debug my day to day problem. Say for example, when I run selenium, and the browser doesn't open up:

1. Does the jar is valid? 2. Safari has selenium plugin? 3. Have custom installation rather than default installation location?

Then the smart you make the guess, the smart you can solve the problem.

Its about removing the un-wanted space ( by making smart guess ) and debugging the right space.

I'm curious, is there anything like this for Android and calls to any Java system functions?

(it should be possible with XPrivacy)

Reply on news.ycombinator.com