OVH servers exploited through shellshock
status.ovh.net
status.ovh.net
At best this is monitoring and playing whack-a-mole on their part which they've done a pretty good job of by the looks.
Also, although I have many things to blame OVH for, they at least take a somewhat proactive stance regarding security, even for the very cheap dedicated servers. They usually contact you when they notice a traffic spike on unusual ports for example.
But yeah... judging by the people I know in real life who own cheap OVH dedicated servers, I'm not surprised at all by these news.
I'm pretty sure 90% of the customers are not professionally trained in server maintenance. I am, and I don't want to do it anyways unless I have a good reason to do it.
The company I currently work for have to full racks in two data centres each, most of which is near idle and it's a full time job for 2 people. Could do away with all the machines and all the associated administrative staff, just use a devops team and shave 40% of capex and opex as well by moving to something else.
But then again, they might not be getting nice lunches with HP and the DC vendor.
The hard part was to set-up all these services, keeping them up-to-date, since no other party depends on them it is considerably easy and almost risk-free. Other than that is not time-consuming at all. Only 2 times in 3 years I had to re-configure a service. Most of the times I just run an upgrade-script on a tmux session and everything runs smoothly (I rarely use pre-compiled packages).
Now, if I had a production-level web application, I would probably set-up as carefully as I could a new VPS and roll my application there, up to the point where I needed to scale.
If it would be a one-man-show I'd probably go with Heroku (since I write sinatra/ruby/rails applications) or similar (possible cheaper) service to avoid spending time on sys-admin. But these service IMHO are still, expensive for projects with no income while a VPS can do all that and mode at once at a considerably lower price.
ps. As a side note. Before I join HN, I though that all programmers were capable of sys-admin (UNIX servers). Then I realized that tasks that seem trivial to me, are considered somewhat difficult for others and vice-versa of course.
If you're asking about the hosting company, they largely don't care what you do unless someone complains.
I don't think DO offers any managed service. They wouldn't do anything to fix your servers. It's entirely on you to keep them updated.
Edit: https://www.digitalocean.com/help/policy/ <- they really don't offer it
I know that many of us that run systems for which availability criteria mean they want full manual control over all updates, but those are the people that either use custom images or know how to turn those automatic updates off.
Security updates are finally pushed hard on desktop users for obvious reasons, with everyone and their mother running cheap VPS's these days the same logic should apply to virtual servers.
Ubuntu, for example, can be configured to run apt-get update && apt-get upgrade automatically every day (using unattended-upgrades), but AFAIK the default configuration only checks for updates, notifies the admin, and waits for approval before actually changing anything.
Even though Ubuntu LTS, Debian Stable, CentOS, etc. are supposed to be stable distributions, it is not unheard of for routine updates to break something mission-critical. For example, the package maintainer might have made some changes to the default configuration. (C'mon, do you really need to make minor changes to nginx.conf all the time?) A daemon might go down and fail to come up again after an update. (apt is particularly annoying in this regard, since it forcibly restarts daemons during an upgrade, and not always in a sensible order. Result: HTTP 502 Bad Gateway.) So unless a human is present to spot any issues immediately after an update, you might be left with a broken system at an inconvenient hour. Which is why the documentation doesn't recommend fully automated updates.
Unfortunately, the set of VPS owners who don't apply updates regularly probably has a large intersection with the set of VPS owners who won't be able to troubleshoot a broken update anyway, so maybe this concern need not apply...
(CoreOS employee)
OVH isn't responsible for what people do with the servers. They provide the initial installation, networking, hardware monitoring and some management tools, but that's it. If they detect that a server is sending too much traffic, they can guess that it has been hacked, so what they usually do is disable it and notify the owner to fix it. But they don't do more, that's not their business. If you want a host that does more for you, it's managed hosting you want, not a dedicated server provider.
But if you receive DDOS attack to your server from outside, they can defend you using network resources.
This is an internal attack, which requires different mitigation measures, and is seen less often in the wild (compromising 500 servers from a specific provider is more difficult than 500 random servers on the internet, and you're pretty much guaranteed that the provider will deactivate most of them after the first attack), so I guess their protection systems aren't as developped against it.
If it's actually your server that actually attack another server, they will shutdown your server and give you a warning. They will let you boot in their recovery os that let you access your file system but if your server does it again, they terminate your account.
No, it's not. They have a proper anti-ddos solution in place for attacks from outside of their network[1].
[1] https://www.ovh.co.uk/anti-ddos/ddos-attack-management.xml
Their VAC protects external inbound but does nothing in their intranet which runs at full tilt 1gbps for most dedicated servers.
http://status.ovh.com/?do=details&id=8038
Tata transit in Montreal has been saturating at peak too.
I'm hoping all this was just internal botnet/DDoS that was ramping up and the issues will go away as they clean things up.
The story and scripts were published on a blog post in case anyone wants to check out a standard botnet attack:
http://blog.mist.io/post/100582053116/anatomy-of-a-shellshoc...
I used one of their dedicate servers for six months and never had any problem with it, however I plan to buy (rent) dozens of big-data servers for my next project, and while my experience so far have been good, I'd like to hear from someone else.
Despite of that, I had several downtime issues with VPSs hosted at OVH. I guess the difference is you can't oversell dedicated servers.
Last year i switched to collocation instead, but still have 1 server with them, things have improved alot lately, now the servers come with ipmi and new ovh control panel is nice.
Problem with OVH are noobs buying their servers trying to save every buck only to realise that the support will not fix anything but hardware issues (loose cable, broken drive etc). Obviously alot of people do not understand that unmanaged means unmanaged
From what I understood, shellshock was caused by poor sanitation of bash variable extension, and so only posed a threat to people who used bash from their public-facing interfaces, hosted bash CGI scripts, ...
Was I wrong to assume that I was safe?
GET / HTTP/1.1\r\n Host: () { :;}; echo vulnerable\r\n User-Agent: () () { :;}; echo vulnerable\r\n\r\n
So even static servers could have been vulnerable. I'm not sure if your server was vulnerable or not, but I think you were definitely too soon with assuming you were safe.
This means any CGI script is affected if bash is the default shell, for example.
So yes, patch everything, or you are taking a bet that there will not be any sufficiently clever attackers in the entire future lifetime the server.
I've seen people try to exploit this by email, hoping that users have ".forward" or ".procmail" files which are processed with bash, or which use it to execute commands, for example.
This has likely happened to loads of VPS and dedicated server providers. One consequence of hosting such a service is that you typically end up with a lot of poorly administered crack houses on your network.
This will save them much time in the future, save their own some time, and save other sites (that will be attached through the compromised hosts some time).
Being a good citizen of the Internet boils down to taking some level of responsibility. I wont put grandma on the internet without malware bytes or avg, etc on her computer. So why should I give her a VPS and tell her she's on her own?
This hack is on Dedicated Server. OVH do not have the right and the access to the system of thoses servers.
By the way, all the clients have been informed Two weeks ago by a global mailling because of Shellshock. The responsability is all on the customer.
That changes quickly if links get saturated suddenly and other customers are impacted.
You should check your TOS. They reserve the right to access your server and if you use any of the default install images it comes pre-configured with their SSH key.
Do you suggest that every hosting company run non-stop dictionary attacks against their clients?
Or even nmapping open ports, non-stop, looking for backdoor services?
95% of the time virtual machines are running unmanaged, and the hosting company has no insight into what is running inside the machines, nor how up to date they are.