Hacked: Investigating an Intrusion on My Server | Hacker News Reader

Hacked: Investigating an Intrusion on My Server

thedarkside.frantzmiccoli.com

100 pointsfrantzmiccoli36 comments

This article was absolutely painful to read. Let me try a different ending:

> Conclusion:

1. Ensure the files that php is running are not writable by the same process. Different app -> different user.

2. Unless you're planning to send emails from the server, firewall output on those ports. If you do plan to, firewall everything apart from that server (you can setup alerts when DNS changes, it's not going to happen often for email hosts)

3. Disable most unnecessary functions and modules. Anything touching eval, str_rot13, exec, and many others should be killed right away.

4. Enable basedir.

5. If you can't handle regular system updates, don't run your own server. If you can't handle wordpress updates, host it with someone else.

6. Wherever you host, make sure your app can be easily redeployed. You can't rely on dates to see when anything changed.

7. If you can't handle system upgrades, don't think that docker or any software is going to solve any of your problems.

I've never worked with PHP before, so basedir meant nothing to me. A minute with Google turned this up:

http://php.net/manual/en/ini.core.php#ini.open-basedir

I have to say, for all the general brain-deadness we hear about from the PHP community, the core developers have made a lot of progress on cleaning up their mess. I don't know of anything equivalent to this functionality from Python or Ruby.

I think basedir like functionality is a good idea, but this is not the languages responsibility, this is an operational concern.

There's too many ways around it if you're trying to do this in a language ... for example, limiting checks on the open() call would be relatively straight forward, but what about people who use syscalls directly from the language? Or what about shelling out to another process?

Chroot jails get kind of close of this in a very broad/blunt way, but back to my original point: This is not the languages responsibility

PHP doesn't have direct access to syscalls. (not without extensions anyway) exec / system / other shelling out functions should be disabled unless you actually know you need them.

This is not new. PHP3 already had it, but not many people actually care enough to use it.

Other languages have LSMs and seccomp (linux), pledge (bsds), chroot (all *nix). It doesn't need to be built into the language itself.

Unfortunately on point #1, the majority of Wordpress deployments are sitting on cPanel servers running suExec. More often than not, the application runs as the only account a customer has access to.

Yep. I used to clean up servers at a hosting company that would get hit with malware and if we had done it like this person did, they would get re-pwnt in a few minutes / hours. Much of the pwnage wasn't even personal, but automated exploitation en masse of outdated PHP scripts like Wordpress, Joomla, PHPNuke, SMF, PHPbb, vBulletin, one of the 10 popular photo gallery scripts, etc etc.

The best bet is to back up the DB and reinstall the PHP scripts clean on a new server. Upgrade them to the latest version in the process. Try not to re-use any files from the old server, and inspect the heck out of any you do.

Suphp + suhosin + open_basedir + SELinux + chroot, jails, or lxc, proper permissions + proper Umask and Apache config + LMD + tripwire + mod_security + owasp rules + clamav + competent sysadmin team + monitoring + alerting will get you most of the way to a safe PHP hosting environment. Not impossible but not something I'd recommend considering how cheap decent hosting is these days.

I think your last paragraph is overstating things a little. If you're providing hosting for other 'random' clients who upload their own plugins, sure, you need to be wildly over-zealous.

I, and probably many others, provide hosting for a few wordpress sites that we admin, and a client has only editor access to.

The way I've got it set up is:

- nginx running as nginx:nginx, handling static files, and passing on to PHP-fpm only for non-upload directories. - each site directory is owned by $clientname:$clientgroup. - php-fpm runs in each dir as $clientname-php:$clientgroup.

So wp-config.php, etc. configuration stuff is readable only by the group, and only writable by the owner user.

The PHP running user can upload media, post new content, but cannot upload plugins, edit any other PHP files, change any site-level configuration, etc.

To upload plugins, or do updates, inside wordpress it asks the user each time for a FTP username/password, which it uses to create a local ftp connection in to the server, as $clientname, which can then write to the plugins directory. FTP is only accessable from localhost, so there's nothing in plaintext on the wire.

Wordpress and plugin updates are done by WP-CLI hourly as a cronjob by the $clientname user.

Each site has its own database, user, and all that too.

All of this is set up by ansible, so it takes me about 3 commands to create a new site from scratch, or 5 to provision it onto a new server.

I've got reasonably defaults set up in php-fpm, so basedir, maxupload, restrictions, and all that. And as much as possible restrictions in place by nginx as well.

I'm working on automatic SSL certs from letsencrypt, which is working on some sites now, which beats the self-signed ones I had before (log-in only to wp-admin by SSL).

It seems relatively good for me. The sites are fast, cheap, and for my clients (small businesses, friends, family, church(s), etc), perfectly adequate.

I understand the full stack, and can keep the very cheap VPS running in less than an hour or so a month.

In the future, I hope that there is something more self-contained and easy to deploy, perhaps like caddy[1].

I'm not a huge fan of wordpress... but it's everywhere, and enough people have heard of it / can do stuff with it. The bus factor is a lot lower than many alternatives.

[1] https://caddyserver.com/

Are your ansible scripts available, as this sounds an awesome setup?

Thanks.

I'll try to at some point - but a lot of it was hacked together pretty quickly to various deadlines, so I'd like to refactor some bits and audit it to make sure I didn't do anything stupid like put mysql root passwords in a config template somewhere... I'll try and post to HN when I do.

Caddy is brilliant. Give it another year and I think it will have a significant share of the webserver market. I replaced about 200 lines of nginx configuration with about 15 lines of Caddyfile. This is mostly due to sane defaults and inbuilt support for Let's Encrypt.

If you have stylistic comments/advices about the "reading pain" you get, they are welcome!

It was a pain because of conclusions / actions rather than the writing style.

Ok ;) sorry for that

In short: an old server running un-updated software was broken into not once but at least four times since the start of the year.

The post describes some steps the author took to investigate and block the attack. It's an entertaining reading, but I'd strongly advise against trying this at home: it is not worth the risk. Reinstalling the system from scratch instead would have been much more prudent.

Yep. "Treat servers like cattle, not pets".

I won't recommend Docker but something like Ansible can be used to make reproducible servers.

Also, remember that while no one likes backups, everyone wants restore. ;)

I think running a Docker container for Wordpress or Joomla installs. If setup correctly with the right folders setup as volumes (i.e. wp-content for WP) and the rest would be part of the docker image. If WP gets corrupted/infected, then starting a brand new image would remove most of the corrupted files.

Fun read. If you ever get hacked, I recommend just destroying the server and starting from scratch and a clean backup.

If you try to find all backdoors and left over rootkits, you will end up forgetting one and being re-compromised.

You can learn a lot by investigating how the attacker got into your system.

Sure, but you shouldn't continue using the system.

One may find the following resources useful for measures on how to secure one's infrastructure:

- Security Guide: How to Protect Your Infrastructure Against the Basic Attacker [0]

- 7 Security Measures to Protect Your Servers [1]

[0] http://blog.mailgun.com/security-guide-basic-infrastructure-...

[1] https://www.digitalocean.com/community/tutorials/7-security-...

Question about this:

It's not possible for me to track all the 0 days for every piece of software and library that my servers run. One of my long running servers could have been backdoor-ed by a 0 day 6 months ago and i probably wouldn't know it. The servers are kept updated but 0 days don't care about that by definition.

What's the best practice here? Should we pre-emptively have our servers rebuild daily just in case a 0-day backdoored them?

> What's the best practice here?

Defense in depth.

Remove the unneeded code so it never executes. If it's needed, block features you don't need. For the rest go one level higher and block operations you don't need. (php.ini controls, file permissions, LSM, etc.) For operations that are allowed, ensure data/code separation to prevent persistence. When that fails, have a very restrictive firewall so that data extraction and attacking other systems is difficult. Have network restrictions again one level higher. (switches / cloud network) If everything fails, have detailed logs to know what failed and when. If even that fails, have backups. If that fails, have company insurance prepared.

> Should we pre-emptively have our servers rebuild daily just in case a 0-day backdoored them?

Depends on your threat model and available money. Are you trying to stop automated malware or state actors? In the first case a good configuration should be enough. In the second you may want to start thinking about how you're going to verify the firmware your harddrives are running. And there's a whole spectrum in between.

I mean, to block majority of php blog/cms malware it's enough to make sure your code is owned by user A, php server runs as user B, and B cannot write to A's files. 0-day or not, lots of stupid automated malware stopped.

> It's not possible for me to track all the 0 days for every piece of software and library that my servers run.

That's why it's nice to run unmodified distro software. Because these guys can track it for you: http://www.ubuntu.com/usn/ (for example, any reasonable distro has similar announcements)

You should consider running this malware detection script[0]. That script is designed to catch malware php scripts. I've used it before with lots of success. You will also want to check the access logs for all php scripts hit in the last few months. Your hacker's spam script is likely being activated/ran by a GET or a POST to that script. That's a pretty cheap way to screen for other compromised files.

[0] - https://www.rfxn.com/projects/linux-malware-detect/

I did get one of these OVH spam warning emails too once. It was for a failover mail server. It turned out that the software I was using (smartermail) wasn't using TLS by default so the link between the failover and the primary server was unencrypted. Spammers often target failover over primary servers. And OVH was interpreting the failover passing on the spam back to the primary server as me sending spam.

Don't delete the outbound mail queue as they have a header in them which tells you which php script sent them.

Also in the httpd logs look for POST 200's to scripts that shouldn't be getting them, and look for long query strings esp with base64 in them. Check those scripts and also scripts messed with by the same user-agent and ip's.

Never just delete anything, always check the contents and write a script that can search and nuke it en masse. e.g. Find | xargs | sed etc.

When I had issues earlier this year I had postfix outbound throttled right down, and a script to shut it down completely if the queue got big. So if I got the monitoring email about smtp going down I'd know I'd missed something ;)

Ps, wp-cli, you need it in your life :)

Thanks for sharing your intrusion story. Although it is always painful for the person to experience, it gives good insights on what happens in the wild. As the author of security tools rkhunter and Lynis, it is still a sign that basic (and advanced) system hardening remains needed for a long time.

The other comments show that there are so many things you could do. But some are more relevant than others. That is something we want to test for in Lynis with upcoming updates. One focus area will be detecting Drupal/Joomla/WordPress installations. If anyone wants to help out the open source project and make the web a safer place, this is your chance to help: https://github.com/CISOfy/lynis

Thanks again for sharing!

Ahhh PHP, where `assert()` `eval()`s strings passed to it.

I wish it was easier to convert "standard" dynamic web pages into static HTML easier. There's one or two Wordpress sites I'd really like to get rid of updating over and over again (which is still less work than actually reading mostly meaningless Changelogs and making a decision if there are security issues that actually apply to me). Probably, my solution to this problem will be "wget -r", plus a bit of sanitizing.

There are plugins for this - Simply static, staticpress and more.

Does }__ appear in your logs? All versions of all branches of Joomla prior to I think 3.4.6 had a problem with serialization that allowed arbitrary PHP execution.

Here are my advice:

1. once compromized, start with a fresh copy. Use ansible to define automatic installation scripts so that the server can be fully reinstalled in a "snap". Use backups of the user data to restore.

2. Automate upgrades. It's a piece of cake with ansible.

3. Stay away from php. There are too many potential loopholes. I now start using go to implement my web servers. Especially if the web site is static. Check Hugo or tools like that. I implement dynamic servers with go and Iris that uses fasthttp.

> I now start using go to implement my web servers.

Then again, you can write buggy/insecure code in any language.

I don't think I've seen any articles about potential security pitfalls when writing web services (or code in general) in Go. I'm actually interested if there's anything surprising out there.

The difference is that we have a better control on what is being done when using a statically typed language. Of course it is always possible to create a vulnerability too.

There is always something to be learned. Thank you for sharing.

Reply on news.ycombinator.com