Insecure/Hacked server: lessons learned

I've recently had an educational (and sleepless) night as a result of having a server hacked so I thought I'd just give a gentle reminder for everyone to secure their machines.

Some details:

  Cost: $53 (bandwidth)
  Machine: Win Server 2003 hosted with Rackspace
  Data Loss: none
  Duration of comporomise: ~3 Days
  Data in: 300GB
  Data out: 330GB
  Data out rate: 2-4% of 2gbit connection used 24/7
  Down time: ~30 hours

Does anyone have any idea what all this data in/out could have been?

most wtf moment: installing wireshark and watching my password fly past in cleartext

This was a dev box I kept for testing. In no particular order here's a list of stupid mistakes I made:

  1. Machine unpached
  2. Apache outdated
  3. MySQL no root pw
  4. httpd and mysql processes running as root(!!! ffs what was I thinking)

I knew these and other things needed to be eventually set up, but you know how it is... there're always a million other things to do :|

My guess is that the box was comporomised via apache. Running MSSecurity essentials yeilded no less that ~50 trojans. Not sure why so many. Maybe was compromised by multiple attackers?

I was running a few sites for friends/acquantances from this machine as a favor, so my next thought was: how do I not inconvenience these people with down time?

Purely by luck I had made a full backup up of /web just 2 days prior so a restore was going to be painless. If not for this, recovering anything from the machine would have been a very painful process since any traffic from Rackspace-->My Home Machine would need to be tightly managed.

So the test box was toast and the image needed to be destroyed (a true case of: 'aghhh burn it with fire!')

This was finally the motivation I needed to secure a machine, especially since I'm just weeks away from lauching my first web app.

These tips mostly apply for *nix/win

  1. Remove any unneeded accounts. Windows has a permissive 'Users' group. My suggestion is to remove any accounts from here
  2. Create a new user group and a separate Apache and MySQL user.
    -- These two users should have strong and distinct passwords
    -- for some strange reason, in windows creating new users default to the 'Users' group. be careful to make sure your new accounts go where you intend
    -- Remove all permissions from above users
    -- Give Apache user read access to /apache, write to /tmp and thats it.
    -- Same deal with MySQL read to /bin, write to /data
  3. Need FTP? Only open the ports and start the service when needed.
  4. Dont run any unnecessary services. Given I'm curious (and a sucker for punishment) I wanted to run my own mail server. Dont. All mail now goes through 
    a Google apps account. This means 2 fewer ports open on the machine and MUCH better deliverability. I was never able to configure my server to deliver mail to hotmail without going to junk.
  5. Enable auto update for win
  6. Stay on top of security updates for Apache. Plan out an upgrade schedule
  7. Revoke any SSL certs which were on the machine and regenerate certs
  8. Change all you passwords for all services (registrar, hosting, email etc)
	

For windows: I was surprised how easy it is to grant more permissions for a user than intended. Double check the permissions from the 'Effective Permissions Tab' in the folder security tab. To double check and for your own peace of mind: log into the machine as Apache/MySQL and see what you can access/do.

Wow this turned out longer than intended. Feel free to add any other hints/tips.

Resources: http://robsnotebook.com/xampp-security-hardening gives a nice general purpose checklist for securing xampp (applies to any apache/mysql install)