Heartbleed Update | Hacker News Reader

Heartbleed Update

blogs.akamai.com

54 pointssafeaim13 comments

> Do you have any evidence of a data breach?

> No. And unfortunately, this isn't "No, we have evidence that there was no breach of data;" rather, "we have no evidence at all." We doubt many people do - and this leaves data holders in the uncomfortable position of not knowing what, if any, data breaches might have happened

I like their honesty - many organizations simply stated "our evidence suggests there was no breach"

Maybe Akamai and other large internet companies (Google, Facebook, Cloudflare etc) contribute now financially and/or with engineers towards openssl development or create an alternative.

This whole thing has been one giant clusterfuck, I myself seen one rather larger alexa top 1000 site being exploited by sessions being hijacked.

Totally agree.

I'm finalizing a tool to scan and visualize the top 1M alexa site URLs to see which are vulnerable - and ~3% (30 000) still are. In the last few days I've observed about ~5% of those getting patched daily (~1500).

Akamai posted its special heap implementation</a> to openssl-dev yesterday: <http://marc.info/?l=openssl-users&m=139723972124003&w=2>. We'll follow that up with help to adapt it to openssl's needs---which are much, much broader than our set of problems. We run 100k machines, sure---but all x86-64, almost all Linux, and all running substantially the same software. We don't use DTLS. We don't use any PAKE systems. We are, in some sense, an easy problem.

The OpenSSL Foundation is trying to help people with those needs and needs varying on every imaginable dimension communicate with secrecy and strong authentication. We should expect them to need several times as many developers full time on that problem as any of the planetary-scale computing companies.

Maybe Akamai should create a challenge just like Cloudflare did...

We did, internally. We got key extraction in an hour, about the same as Cloudflare---so we save $10k and rotate exposed keys. Since we renew most certs on an annual basis, all our easy-to-rotate certs were flipped months ago. The outliers are weird custom stuff, or two-year EV certs.

You ever try rotating 400 EV certs in a weekend? Neither have most Certificate Authorities. They say security is people, processes, technology. Our tech worked well---but not so well I can't wish it was better. Our people did awesome, stalwart work. But the PKI industry processes are due for some serious reconsideration.

I can't tell you how badly I want TACK or DANE or CT live and working right now.

What would that solve? We already know that it's possible to compromise the private key using this bug. The problem is that no one knows whether attackers have actually done so, since very few organizations will log all the data going in and out of their servers.

Does it matter? Once you know they might have, you have to assume they did. That's why the Cloudflare challenge was watched so closely.

It seems interesting that Akamai protected itself from heartbleed by implementing a custom allocator, which was in some respects at least part of what caused heartbleed.

Well OpenSSL didn't implement their own malloc exactly, they used some weird indirection to try and speed it up on some platforms, which may have bypassed protections that some versions of malloc have enabled (guard pages).

Agreed, it's a loose analogy at best. The juxtaposition just struck me, mainly because it could be viewed as singling out the real difference, which is how the code was handled.

In some sense, yes, the special free list caused it. In another sense, the missed bounds check caused it. In a third sense, the lack of proofs of safety, or informal code review process, "caused" it---that one's harder for me to argue.

My own preferred sense is that mixing network code with soft real time performance requirements with crypto in a single library, single process, all in C---maybe that caused it, and will cause problems for any channel-oriented crypto network system. Imagine trying to mix GnuPG with high performance networking! Boom.

My preferred tools for thinking about what causes accidents like this are Leveson's systems-oriented frameworks, explained in Engineering a Safer World. The text is available free from MIT Press, I believe. If you're responsible for the safety of a planetary computer system, you should read it and it's principal competitors.

And if you do, Akamai Infosec is hiring.

I'll keep that in mind, Brian, although my reading list is long, and not getting any shorter.

Reply on news.ycombinator.com