But it comes down to:
- Take existing server down immediately. I'm assuming it is not on an isolated network -- so this should really be a priority.
- Prep a new patched server (with a smaller attack surface and updated security credentials)
- Postmortem the old box on an isolated network. Try to understand how the attacker got in. If necessary, get security professionals involved.
-> disable root being able to login inside your sshd_config file. Make sure PermitRootLogin no
-> rename the root account too so if they are using an exploit based on user authentication then perhaps they won't be able to elevate to root.
-> disable password based logins and go to cert based auth. This will shutdown brute force attacks.
-> lower MaxAuthTries in sshd_config to something like 1 or 2 to help slow down attackers.
-> change the port you listen on. A little security through obscurity while not very effective might slow future casual port scanners that are testing a single port. In practice I've seen this really eliminate a lot of reconnaissance or farming type activities.
-> Make sure you're openssl and openssh are at the latest stable releases.
-> If the perp is coming from the same IP, you can use an iptables rule to block the netblock. Again, not perfect but may help slow things down.
-> grab shell history of all the user accounts on the box. Example ~/.bash_history. This is more reconnaissance but may be helpful if they are sloppy - you might see what they were doing on the box.
-> look for any modified files or new ones. Obviously logs will show up in the list but you're looking for things that should not be there. Example: find / * -mtime -60 -print where 60 is how many days ago you want to go back.
-> look at chron file to see if any timed delay bombs exist.
-> look at ps -aux output for any running processes that don't make sense
-> look at iptraf for any suspicious traffic to IPs you can't reconcile.
Good luck!
(Just kidding.)
command history is also easy to alter if you know what you are doing.
Re the command history, is there any way around them wiping it - e.g. piping all .bash_history entries to an append-only store?
You can also use a global bash configuration that would log all commands [1] entered into any bash shell to a central log, which could be shipped off-server simultaneously.
[0]: http://rkhunter.sourceforge.net
[1]: http://askubuntu.com/questions/93566/how-to-log-all-bash-com...
However, rkhunter has additional logic to specifically seek out rootkits and malware-like behavior, and is more specifically targeted to system file modifications. Combining it with unhide (to compare actual processes running with those visible from userspace) provides a reasonable assurance that nothing nefarious is going on.
They're all part of a spectrum, however. I use scanners like aide alongside rkhunter as well, but I'm the sort of guy that will spend a day tuning the config of aide to avoid constant false alarms.
I'm curious why you think owning a DC makes you less qualified to respond? Presumably because someone that senior is less in touch with day-to-day security operations?
You could also get looped into tax evasion if the person you paid doesn't report the bug bounty income. You can still maintain anonymity by not announcing the person who found it, but it's not a good idea to give money to anonymous people for work.
The few times we've been in this position, its always been someone's desktop that's been compromised. If you don't have a strong desktop security policy, a UTM, traffic inspection, key encryption policy, etc and you store keys or passwords locally, then this becomes a risk.