OVH acknowledges back office security issue | Hacker News Reader

OVH acknowledges back office security issue

pastebin.com

58 pointskerneis24 comments

Translation from french :

Hi, To change your password, you have to go on ovh website and ask for a change with a given id. An email is sent to the email address associated with the id which includes a unique url. This url is randomly generated with 21 characters, generated from 3 different algorithms for randomess generating each 7 characters. The client who receives the email can then click on the link and get a new password for his id. A confirmation email is sent announcing the change of password. In all emails sent, ovh includes the IP of the person who did the request.

This is a procedure that was set up 7 years ago and hasn't changed since.

On april 26th, we have discovered internely an issue with the function that generates the 21 characters. Two of the three random functions were generating a not-so-random string. It was hence possible to ask for a password change and then to brute-force the password reset url. This problem was found by one of our developpers on apr 26th at 11:03:14 and was fixed at 12:54:13. The origin of this problem was linked to the rand() function used in this part of the code that hasn't been patched to the same level as the rest of the code when we activated the script execution cache (?). We have replaced the old function returning 21 characters from 3 "random" functions with a new function using two really random functions each providing 64 characters.

We have then started searching in our databases to check if the vulnerability had been exploited, and when it could have been. To do that, we have looked at 3 years of history of password resets. We are authorized by the CNIL (commission nationale informatique et liberté, french agency regulating databases containing personnal data) to archive and use all logs for the last 10 years, specifically for this kind of issue.

We have found 3 active ids which had a password reset done by brute-force. These three occurences were targeted attacks against a part of the "bitcoin" community that uses ovh. The hacker seems to have found the flaw on apr 23rd at 22:00, and did a lot of testing for one hour to make his method work. At 23:00, his method was working and he has hacked the first id, then the two others ids the following day, still for bitcoin related sites. We have been in contact with these clients but the quality of our exchanges did not allow us to get enough information that would help reveal an vulnerability on our side. Our developpers have in a completely independant way found the issue mentionned above, and only then have we realised that there was a link between our flaw and these 3 customers. We certainly have something to learn here about the best way to ensure dialog with clients about this kind of issue.

It took us some time to communicate on the subject, because we have quickly realised that the impact was very limited (3 customers) and we wanted to take enough time to verify everything in depth and make sure that this had not impacted any other customers. Having finished our search today going back three years, we can now assert that no other customers have seen this issue. We are going back 10 years now, but the probability that this happened ealier is null.

I think that despite the small impact for our customers, it was our duty to inform everybody of this security incident that we had to deal with last week. We have added a code-review process on some very old code that hasn't been rewritten for years, to check that there are no other similar issues. We are also trying to see how we can improve communication with clients for this kind of incident, knowing that 2 of the 3 customers are actually customers of our affiliates/branches.

TL;DR : Yes, we had a security vulnerability allowing a password change using some complex brute-force procedure. We advise all clients running critical services to limit access to the manager from some IP only. Yes, 3 clients of the bitcoin community have been impacted by this security flaw. It is very important to read the emails that ovh sends automatically, especially the notification of password change emails, when they aren't initiated by you. If you see a request for a password change that you haven't initiated, you should call our incident support 24/24 which will block your account until everything is clear. No, our clients database hasn't been compromised. No, there was no impact on other clients than these 3.

We are deeply sorry for the three customers that have been impacted, and invite them to contact our commercial team (in french).

Regards Octave

Really good article on the pitfalls of a weak random number generator. I think it's been on HN before: http://www.cigital.com/papers/download/developer_gambling.ph...

For me, this was the article that made it crystal clear why you cannot use current time as a seed for a RNG.

"script execution cache" - perhaps something like APC that would cache the PHP opcodes and make a random function less random.

Perhaps something they were using to seed the random number generator was being cached, leading to the same set of "random" numbers?

"We certainly have something to learn here about the best way to ensure dialog with clients about this kind of issue. [...] We [...] invite them to contact our commercial team (in french)."

To ensure dialog, why not start by being able to communicate with clients in English? (or their own language, for that matter).

Hmm... rate limit, anyone?

Just thinking here. If you sadly used just 26 characters in this space (rather than the 64 you can easily get in a URL), 26^7 ~= 8B codes. So, even if you perfectly got the first 14 characters, you'd have to brute force in the billions of attempts here. If you even more sadly did just hex characters (16^7) you're down to ~250M.

However you cut it, someone had to hit up the reset API a bunch of times even in the worst case. Regardless of the entropy and key length used, if you're going to allow that many attempts, someone will get in.

That and shame on anyone for assuming a basic math lib rand() function creates security grade entropy.

A pastebin link is not a source worth reading.

Here is what I found as more official, the oles@ovh.net account is often used for official announcements: http://forums.ovh.net/showthread.php?p=548183

tl;dr: Bad random string generator in password change URLs, 3 customers affected, all Bitcoin related.

edit: OVH, if you are reading this. The 4 character string for receipt protection is creeping me out ever since I saw it. Why not simply use a longer string?

Thanks for the link. I've been looking for the list archives on http://ml.ovh.net (as advertised in the email itself) but the domain doesn't seem to exist anymore; hence my (unreliable) copy on pastebin.

I'm 6 for 5 at the moment - of the 6 companies I've hosted pet projects with over the years, 5 have been compromised at the control panel / infrastructure level while I was a customer:

  * Layered Technologies, dedicated OpenBSD (Sep 17 2007) [0]
  * Hetzner, dedicated Linux (Oct 21 2010) [1] 
  * Dreamhost, shared Linux (Jan 19 2012) [2]
  * Linode, VPS Linux (Apr 15 2013) [3]
  * OVH, VPS Linux via reseller (Apr 23 2013) [4]

  [0] http://what.entwinedstudios.com/?p=21
  [1] http://mybroadband.co.za/news/internet/16021-Hackers-hit-Hetzner.html
  [2] http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-passwords-due-to-security-issue/
  [3] http://blog.linode.com/2013/04/16/security-incident-update/
  [4] https://news.ycombinator.com/item?id=5632479

Food for thought.

There we go. I suspected that Oles was misinforming/lying to the community when he initially denied the existence of a security vulnerability, and I was right: https://news.ycombinator.com/item?id=5627487

When you think about it, this whole situation with the hacks of Linode, OVH, etc, is completely insane. Bitcoin has become so successful and valuable, that bitcoins thieves are compromising core Internet infrastructure that so many companies rely on.

People say "use more serious/professional hosters", but OVH is the world's largest server host and is very much a "professional" service. I bet that within 2-3 years we will see more and more stunning hacks of other hosters like Amazon EC2, etc.

The flip-side is now OVH is more secure than they have been in the past. Nothing like a pot of gold to motivate people to try to break your security. This is probably more effective at exposing weakness than paid pen-testing.

Not that you ever want customers to lose money/data/whatever in the process, but it's not all down-side.

It's not however a high-security host. If you need security you should at least be using a provider that meets the security provisions of PCI, HIPAA, FISMA, etc.

If banks and government agencies aren't using it, then it probably doesn't have the security level needed for high-value bitcoin wallets. There's a reason you have specialist data centres that focus on sectors like finance.

From the post on the french forum: they were generating a 21 characters string from 3 different sources of random (each source contributing 7 characters to the final string) and it looks like that among these 3 sources, 2 of them were flawed and were not really random. They say they fixed this flaw by now generating 64 characters random strings from 2 sources of random which they think secure.

How hard is /dev/urandom? Or without that, read /dev/random every so often and mix it with SHA1?

TL;DR: Here's what happened. When a password reset is submitted, the user receives a random URL by email containing a random 21 letter string. The algorithm used was bad, so the random string could be brut-forced.

The OVH team researched 3 years of password changelogs and found 3 customers who had been brute-forced that way, all in the "bitcoin community"

"oles" is Octave Klaba, the CEO of OVH. I'm quite impressed by how transparent they are in dealing with this issue, and the degree of technical details they are willing to provide.

Vision of the future: Hosting companies which do not provide insurance against such losses, subject to intensive security auditing, justifiably lose business.

If you decide to stash your life savings into your local swimming pool's locker, don't be surprised when they are taken away.

For $40~$100/month, OVH offers adequate securit for your regular daily deals site or micro-SaaS. If you expect the level of security that comes with PCI/HIPAA compliance, expect to pay two orders of magnitude more. At some point, someone in the Bittcoin community will bite the bullet and pay the price, and that will be a huge competitive advantage.

This. OVH is a budget provider. While this doesn't excuse them from fault here, you definitely get what you pay for in this case.

Not sure how relevant this is but there is a huge amount of spelling errors in this email, which makes it look all the more authentic but very unprofessional.

Also surprising that this email (if the pastebin is authentic) appears to have been written by a mail client from 2006 suffering from at least one public GnuPG-related vulnerability.

Octave Kabla, aka Oles is native from Poland, his french has always been quite approximative (like my english, I guess).

Oles, what a CEO. I have had direct contact with him, as OVH is a partner of one of my projects, and it's not that easy to find someone so passionate, humble and responsive as he is.

Can someone proficient in French provide a tl;dr?

I have been thinking about renting a server from then so this interests me greatly.

21 character reset was generated via 3 separate functions each generating 7 characters each. 2 of these were old and weak, resulting in a URL that could be brute forced.

Once discovered, they analyzed 3 years worth of logs to see who was exploited (not clear how they matched it..maybe scanning for brute forces?) and found 3 sites all related to bitcoin.

The weakness in the algo has been resolved by using a 2 real random functions to generate 64 random characters

Reply on news.ycombinator.com