What I learned from a hacker attack
grobmeier.de
grobmeier.de
At step 3 one should be thinking "I now know nothing about this box, what's installed? What's modified?" You can't know if its been modified or rooted.
While I'm certain its possible to replace software in the system piece by piece until you trust it again, but that's much harder than what I would say is your only option:
Wipe the disk, put a a new install on it and restore your sites from backups.
EDIT: BTW this is also why you should be backing up your data and config files, not a bare-metal image of the system. OS installations are quick and the bare-metal image is just going to put the problem back in place. Remember, you probably do not know exactly when the break-in actually occurred.
If hosting companies wiped a server every time one of their customers sites got owned the servers would hardly be online at all.
Anyway, I have already mentioned in my post that I was an idiot to forget about the installation. Sure, and that is what I have learned about: not to forget again.
That said, hacking is non acceptable no matter what mistakes the owner of a website has done. I really have no respect for such people.