PHP-FPM remote code execution bug exploited in the wild
github.com
github.com
[1] https://nextcloud.com/blog/urgent-security-issue-in-nginx-ph...
Then dont share your links publicly.
If configuration requires a path to get further than a canned reply from nginx (403, 404, static page..), then it should reduce attack surface a lot. You should not be able to get anywhere near php without the path.
Put your services, on a shared domain name, only yourdomain.com and under a sub-path, like yourdomain.com/thisISAlmostLikeaPassword/nextcloud the subpath is hidden by TLS, unless you make it public by posting it on the internet. And also if you arent careful, like using google "auto-suggest" or just using any Google products, then they will at least know about your path.
Why not just add real HTTP authentication to the site instead?
One should always be wary of password-like mechanisms like secret paths, secret ports, etc. since none of these things are made to be secret, and could be disclosed by something unforeseen. (Paths, for instance, are saved in your browser history/cache, your HTTP caching proxy, if any, and also in the server’s access logs.)
But you see, for what we are discussing here, you could have exploited it even without authenticating, and especially it would have been easier for scanners to find it and exploit, if it was on its own domain.
Defense in depth.
For some services, yes I do basic http auth, besides their own shitty auth.
Downgrade attack, would not work since I use HTTPS Everywhere, and once my browser has visited the site it refused to downgrade - that header is set.
Thanks to letsencrypt "now" (for some time, I know.. but I wanted to do this way before they allowed one to) allowing wildcard TLS certs, I host the above on a domain which doesn't have a single public IP DNS entry, yet has full proper "validated, browser approved" TLS cert.
IOW, I fire up my zerotier client on my phone, open brave, put the URL in, and off I go. https, and for my eyes only.
It's great!
Where people set wildcard DNS this means passive DNS reveals typos, as well as such "hidden" services. wwww.example.com and ddd.example.com are common typos for www for example whereas int-test.example.com is maybe interesting to black hats.
I mitigate code execution worries by running all of my services in individual LXD containers. They're all using isolated user namespaces (unique mappings), and are firewalled away from being able to access my internal network. The data is bind-mounted from a ZFS filesystem which is backed up by the host and uploaded to BackBlaze. The containers themselves are also snapshotted by ZFS. Thus, I think the risks of exploits being able to do much damage are greatly reduced.
However, there is still a worry about information disclosure. Yeah, NextCloud can only access the documents it manages -- but some of those documents are somewhat sensitive. I don't know what the ideal solution for this would be (a wholly separate NextCloud instance just for accessing the private stuff? But what if your family needs to access them?). My main worry when hosting NextCloud was that I am entirely trusting the safety of my NextCloud-stored data to an authentication flow that they wrote themselves in PHP (and has had pretty ugly flaws such as silently disabling 2FA or letting you bypass it by clicking "cancel".)
This is a feature. Besides, you can send friends and family a QR code to connect to your WireGuard VPN. It isn't perfect, but it beats having your personal data stolen.
I feel like throwing everything behind a VPN and pretending it is secure is a crux.
Several famous break-ins over the last ten years have hypothetically been on the inside of that wall.
Better to isolate services from each other limiting cross service jumping, than to build security around a single point of failure.
Service isolation alone doesn't help when my private data is potentially exposed by this exploit.
I agree that it is better, but let’s not forget that building security around a single point of failure is still an improvement, that is simultaneously both high and low friction.
Bad: everything exposed to the internet
Good: everything behind a VPN
Best: Every application on its own micro-segment with access control up to the application layer to restrict all forms of access beyond the bare minimum of what is required.
Perfect is the enemy of good.
> Google has gone the opposite direction
Google scale solutions are great for google scale organisations. They don’t always scale down very well.
I hope for the operator team that they have good tool support to help administer all the access controls. Over time and across large organisations there are going to be a lot.
The even larger challenge must be auditing all these access controls. Services change, and if a connection is not required anymore, it should be painless for its operators to get rid of the corresponding access control.
For a home network simple multi-port knocking should be enough (combined with --ctstate NEW even better). If port knocking or SPA is too cumbersome then at least consider limiting access based on GeoIP, block tor exit nodes, etc (ipset is pretty amazing).
This can be applied to any service on your network btw, including Wireguard. I like knowing that a portscan of my network shows nothing open. I don’t end up on a list that gets used in the next ‘spray and pray’ attack.
Disclaimer: I’m not advocating this for serious use due to replay attacks and IP spoofing via a VPS. This is for home network protection (a boring Class C non target).
Second, bugs are found every day, and your best bet is to use automatic security updates provided by your distro. Yes, if you host anything, you need to be a bit of a security guy and a small amount of paranoia won't hurt. But to say you must not self-host for security reasons is a gross oversimplification.
This is a gross oversimplification and straw man argument. You can self-host securely.
Almost everything has SFTP built in anyway now though, it's only a matter of time before OSes other than Linux based ones integrate it into the shells and then webdav won't matter so much.
seaf-fuse.sh start /mnt/seafile
set $path_info $fastcgi_path_info;
My old configuration used the `$fastcgi_path_info`, and the new one uses the `$path_info` variable, so I got the following error while starting nginx:
nginx emerg unknown "path_info" variable
[1]: https://docs.nextcloud.com/server/17/admin_manual/installati...
> Solution
> On October 24, PHP 7.3.11 (current stable) and PHP 7.2.24 (old stable) were released to address this vulnerability along with other scheduled bug fixes. Those using nginx with PHP-FPM are encouraged to upgrade to a patched version as soon as possible.
> If patching is not feasible, the suggested workaround is to include checks to verify whether or not a file exists. This is achieved either by including the try_files directive or using an if statement, such as if (-f $uri).
However, that means anything utilizing php-fpm in this version remains vulnerable, and it's just unknown if or how apache + php-fpm, or other reverse proxies for php-fpm are vulnerable - right?
So while I don't need to panic right now, I'll certainly have to take a look at our setups running php-fpm on monday.
Assume your systems are compromised and act accordingly.
Soo you use microservices? Yes! Sooo why can I not find more then one database?
Worst-Case, they can try sending some spam mails or DDoS-attacks, in which case the hoster would zero-route/force-stop them as soon as that's detected. And then I'd have to rebuild them with ~30 minutes waiting for teraform.
So yes, I'm going to act accordingly. By not bothering on a sunday, because the systems are properly isolated and there are procedures in place.
Unfortunately, too many people still copy & paste three-liners from random blogs and call it a day, often overwriting the safe defaults provided by their distro, er, I mean, Debian/Ubuntu. (edit: The RPM world is a whole different beast. When you install typical LEMP components on CentOS 7, both MySQL and Memcached listen on all interfaces by default. Seriously?!)
If you're spreading one service across multiple servers, you can spare the few seconds to open up IPs/ports. The default should keep things moderately secure on a single host.
And it's the config settings every blog ive ever seen about nginx + php-fpm said to use. So I think a lot of sites are vulnerable right now.
[1] https://github.com/mail-in-a-box/mailinabox/issues/1663#issu...
That's not to say another similar exploit might have worked a different way. Luckily that bug is patched now.
E.g., if you follow the "PHP FastCGI Example" from nginx.com, then nginx would protect you from this vulnerability in PHP-FPM:
* http://web.archive.org/web/20150928021324/https://www.nginx....
Here's the current version of the page, which seems to have the same info as the archived one above:
* https://www.nginx.com/resources/wiki/start/topics/examples/p...
(I think it used to be at another URL prior to the involvement of the marketing department in 2015; not sure if it's worth finding at this point, because the bug is not even in nginx in the first place.)
Also: Am I secure if i run PHP 7.2.24 or do I need to change the configs?
[1] https://bodhi.fedoraproject.org/updates/FEDORA-2019-187ae312...
After this, you can start appending ?a=<your command> to all PHP scripts (you may need multiple retries).
I would say it might be better to just test against a localhost server in a VM to be on the safe side though.
It's time to upgrade if you want to stay secure: https://www.php.net/supported-versions.php
The vast majority of PHP 7.0 installations don't use FastGCI and don't use nginx but Apache simply because people used 'apt install php' (or 'yum install php') to install it.
So imho, the impact is very limited.
Do you have a source for this?
Why are you running php-fpm? Do you need to separate request's processes? The speed benefits of php-fpm are part of php 7 so using php_mod is faster now.
Because running just nginx is more convenient than nginx + Apache, where Apache is only used for mod_php. For me anyway. (I only use nginx + php-fpm for a Wordpress instance; I have tons of stuff in other languages running on top of nginx too.)
Not sure how common that really is, I've personally never set things up like that and just use nginx + php-fpm and don't know anyone that still uses apache with mod_php.
Our newer applications are using litespeed instead, and we've found it to be significantly better. You basically get the features of a nginx + apache + varnish stack in a single easily managed service and with better performance too.
For light scripts this is far superior to fpm. On the other hand, always loading php does have it's downsides too as memory consumption can get quite high depending on the number of threads.
This is was also the reason for the fpm hype a long time ago: don't waste memory on php when php isn't needed. It had nothing to do with it running php faster.
What you should choose depends on your need.
The "overhead" of communicating via CGI to a PHP process has nothing to do with the speed of execution of the script itself.
> For light scripts this is far superior to fpm. On the other hand, always loading php does have it's downsides too as memory consumption can get quite high depending on the number of threads.
It's not far superior as the "overhead" of CGI is negligible in the real world. Plus you can pool processes for better scaling. Also, if you are using prefork with mod_php (which is the most probable scenario) it means you are forking an entirely new Apache process and not just "loading PHP" with each request.
> This is was also the reason for the fpm hype a long time ago: don't waste memory on php when php isn't needed. It had nothing to do with it running php faster.
It's not hype, because for a long time, mod_php required prefork because it was not thread-safe (even now it's still a pain to manage re-compiling PHP to be thread-safe for mod_php + Apache)...which means you could not take advantage of mpm_event or mpm_worker.
I've never, ever witnessed that mod_php came close to be fast, let alone faster than PHP-FPM. There's more work to be done in order to prepare everything needed for Apache to pass the data to PHP executable once it embeds it within its own process. Once opcache is up and running, PHP-FPM blows mod_php away (and there are tools to warm up the cache prior to letting the php-fpm node go live).
> This is was also the reason for the fpm hype a long time ago: don't waste memory on php when php isn't needed
I've been present when the "hype" as you called it hit. It had nothing to do with memory as much as it did with scaling. Added benefit was the ability to have PHP-FPM act as a multiplexer towards certain services (database to name one).
Today, there's no reason to use Apache and mod_php. It's slower and worse by definition. It can't be faster. If you receive results that show it is faster, you're either testing it wrong or your PHP-FPM runs on a raspberry pi.
location / {
try_files $uri $uri/ /index.php?$query_string;
}
You perhaps might not be vulnerable if you use an internal directive inside your .php location block.
This is patched in 7.1.33.
> fastcgi_split_path_info ^(.+?\.php)(/.*)$;
If I'm not using this feature of PHP, what can I put in this config value to prevent the exploit from working?
The try_files config mentioned on the page mitigates the issue.
This is also precondition, and fortunately it's not included in standard fastcgi_params files.
PHP security has improved markedly over time (especially app security, not just the runtime), but it's still.. well.. stuff like this. This time around I'm lucky that the sole app I run was using Apache
> I don't think for example I've heard of a Python based SQL injection in many years. Stuff like that seems to still crop up regularly in PHP land
This is an extremely subjective statement based on your personal experience of what you heard and didn't. As such, it's not verifiable and not useful. What is useful is to know that, obviously, PHP, as well as Python, has SQL implementations that eliminate injections for decades. And as in Python, there could be people that ignore it and stuff query params directly into strings. This has nothing to do with anything but these people being ignorant. There are of course tons of web apps in PHP, much more than in Python, so among them inevitably would be crappy ones. If you run one of them, do take precautionary measures.
The parent comment explicitly made this distinction
I'm, not responsible for any public-facing webservers any more. My life is much better.
[Edit: I reckon using a well-designed functional language might reduce the risk - like, PHP was never designed at all, it grew by accretion]
If you dont practice defense-in-depth, you get rekt eventually.
I just checked my installation, its safe, since I didnt follow their "recommended" settings at all.
Since its a PHP app, what I do is put an extra nginx proxy infront of it, so there is nginx 1 (this one runs in its own lxc container with seccomp and has not even a /bin/sh in it, only executables and libc required for nginx, its only a network reverse proxy) - nginx2 (special instance for each php app, also same as nginx1 container with _nothing_ in it, but also a bind mounted ipc_socket which is also mounted to php_app container.
So the php_app container and nginx2 share only 1 ipc.socket bind mounted from the host. Now then, in the php app container, there is also lack of /bin/sh or anything else, not even the package manager, only what php-fpm requires, and the nextcloud .php scripts, are in there. _nothing_ else.
So even if this exploit would have worked, there would still be nothing standard to run in the php-app container (like /bin/ls), except the php scripts themselves, though thats bad enought to steal my nextcloud data.
Only if you looked up their instructions for nginx did you get burned.
We have a fair bit of legacy stuff hanging around, all nginx+php, none of which appear to be vulnerable due to consistent use of try_files.