Hacker News Reader

Comment by atmosx | Hacker News Reader

I manage 2 VPSs. I'm responsible for security patches and everything. If shit hits the fan, it's because of me not because of my VPS provider :-) That's what VPS is all about.

View on HN

True. However, since most clients (especially those that don't maintain their servers well) use standard images provided by the hosting provider, I don't understand why those images don't come with automatic security updates configured by default.

I know that many of us that run systems for which availability criteria mean they want full manual control over all updates, but those are the people that either use custom images or know how to turn those automatic updates off.

Security updates are finally pushed hard on desktop users for obvious reasons, with everyone and their mother running cheap VPS's these days the same logic should apply to virtual servers.

True, you can select for example "Ubuntu 14.04 LTS" and not be protected against Shellshock nor Heartbleed by default.

Automatic updates for server software can be tricky.

Ubuntu, for example, can be configured to run apt-get update && apt-get upgrade automatically every day (using unattended-upgrades), but AFAIK the default configuration only checks for updates, notifies the admin, and waits for approval before actually changing anything.

Even though Ubuntu LTS, Debian Stable, CentOS, etc. are supposed to be stable distributions, it is not unheard of for routine updates to break something mission-critical. For example, the package maintainer might have made some changes to the default configuration. (C'mon, do you really need to make minor changes to nginx.conf all the time?) A daemon might go down and fail to come up again after an update. (apt is particularly annoying in this regard, since it forcibly restarts daemons during an upgrade, and not always in a sensible order. Result: HTTP 502 Bad Gateway.) So unless a human is present to spot any issues immediately after an update, you might be left with a broken system at an inconvenient hour. Which is why the documentation doesn't recommend fully automated updates.

Unfortunately, the set of VPS owners who don't apply updates regularly probably has a large intersection with the set of VPS owners who won't be able to troubleshoot a broken update anyway, so maybe this concern need not apply...

We're working on the auto-update model for servers at CoreOS. If interested, you can read more about our update philosophy here: https://coreos.com/using-coreos/updates/. We think that this is one of the best ways to secure the backend internet, similar to how auto-updating browsers moved the front-end of the internet forward.

(CoreOS employee)

As I understand it, server updates run counter to the idea of servers as static, externally configured fixtures ala cloud formation or its competitors. To merge the concepts you'd need to have aws or whoever manage your cloud formation templates and automatically re-deploy whenever they make a change to them for security reasons... I think maybe our tooling isn't quite there yet, but I'm sure its coming. Disclaimer: I don't have a very deep knowledge of cloudformation, so I might be talking out of my ass.

This can be difficult as some updates can break applications. PHP is problematic that way for our customers on servers that we manage for them. Its not as bad as it used to be (e.g. register_globals), but updates can break customer apps. So if you automatically apply them and their web server breaks, who does the clean up ? The web developer who got paid to create the site but not maintain it ? The VPS provider who gets paid to just host the image ? The customer from their POV (and these are not technical people) will often say, "but I didnt do anything to break it. You broke it! You fix it!" Also, as there are (typically) no regression testing, breakage can be silent and undetected for some time.

I have some vps with digitalocean, I haven't logged in to them in a long time, even after shellshock surfaced publicly. about aftter a month later I checked to see if I am vulnerable through shellshocker.net test and it was negative. I am guessing my provider did something to fix it.

> ... vps with digitalocean ... I am guessing my provider did something to fix it.

I don't think DO offers any managed service. They wouldn't do anything to fix your servers. It's entirely on you to keep them updated.

Edit: https://www.digitalocean.com/help/policy/ <- they really don't offer it

I know they dont, I didn't say they do. But my non up-to-date Ubuntu server wasn't vulnerable when I checked about a month after shellshock vulnerable went public.

Have you considered you may be using a software stack that is not (obviously) vulnerable? Not every software is vulnerable by default on all URLs, just the ones that wind up calling out to bash in a specific way.

If I am not wrong the vulnerability test on shellshocker.net tests bash directly. Either way I could very well be wrong and I might have missed something.

Did you run the system test from the website?

https://shellshocker.net/#systemtest

Yes that's the one I was talking about. Ran the test before and after updating. It was the same (negative). I ran the same test on the relative up todate home computer (Ubuntu) and it was vulnerable on some of the tests. This IS weird.

No, you would need a vulnerable service running on the VPS in order to exploit it.

The scary possibility is that someone hacked you, backdoored the server, and then patched the vulnerability to keep others out.

So somebody used shellshock to break in, then plugged the hole to prevent others from accessing the server, and is now using your server for... whatever?

Maybe somebody other did close the vulnerabilty for you. Maybe the same guy who is lending your server for DDOS attacks now.

Ubuntu, for instance, has unattended-upgrade (that it defaults to on), the premise being that it will forcefully install and use critical security patches. Ideally you'd have done this yourself ASAP, but if not I have to think that unattended-upgrade came into play at some point.

That must be it. I can't think of any other way it could be patched. I didn't know about unattended-upgrade before.

unattended-upgrade in default mode does _not_ update anything. Neither on Ubuntu, Debian or any other distribution.

Default means "the default when you install Ubuntu", not the unconfigured state of that process. For kernel and in-use things it would still require a restart, however maybe at some point DO forced a restart, even if unrelated.

Reply on news.ycombinator.com