My heart is ok, but my eyes are bleeding | Hacker News Reader

My heart is ok, but my eyes are bleeding

blog.leafsr.com

62 pointstptacek26 comments

"Naming a length variable ‘payload’ is borderline sociopathic"

I think it further indicates that not enough review of the code was done before it was integrated. Dumb variable names like that should be caught at review time.

I suppose the lack of interested reviewers is a decent indicator of how many people/companies actually cared about the heartbeat feature.

The simple statement that was given here: http://www.tedunangst.com/flak/post/analysis-of-openssl-free... that the developers just had to test with normal malloc to detect the error is a little to simple for me. I can't see this "simple" solution.

Testing is utterly hard. It means that you need creativity, something that many lack or are to lazy to. Many programmers are to lazy to think about testing, I fear. That is the reason, there exist some very limited number of really good software testers in the world.

When you want to find that bug in testing (taken, you even have the super-malloc), you first have to come up with the idea, to send a falsified message length for exactly this message type. Without this idea, even the super-hero-malloc would have no chance at all to give any indication of a bug.

AND when you have such an idea, you would find the bug, even without any super-hero-malloc ... any malloc or any freelist would suffice, because the result would show you, that something bad is going on.

So the focus on the self-written allocator seems to get a little biased in this discussion.

Yes, when writing security relevant libraries, you should be really, really careful and you should take any chance you get to make it more save. And it might have in some cases increased the chance of finding the bug in production, when just using the standard malloc (with security mitigation active) -- but I really doubt, that it would have increased the chance in testing.

I also must say, that really many successful software products use their own specialized allocators -- the reason is simple: Because malloc is so unspecific, it is not the fastest beast around -- and many specialized allocators have their own security mitigation build in.

A last point: Google also wants (or wanted) to switch to OpenSSL. The reason: Speed. So to blame the OpenSSL guys that they want to make their library really fast, has a little bad taste. Everybody wants speed -- even those that claim other after something has happened. My thought for you: What would be the benefit, if OpenSSL would fail because of bad speed, nobody would use it but an other product -- and exactly that product would have the trouble we found now in OpenSSL (or worse?) .....

Computer Industry can sometimes be unforgiving. How you do it, it is wrong.

Agree with most of this except for "and many specialized allocators have their own security mitigation build in." This is generally not true in my experience [1] when it comes to unlink checks, guard pages and the like. Often a generic homegrown allocator will call mmap/VirtualAlloc and chunk the page(s) up into a doubly linked list or simple arena allocator.

Sometimes a default generic allocator will have design properties about it that make exploiting certain bug classes difficult (e.g. WebKits RenderArena and Use-After-Free). There are exceptions of course, see Chromes Partition Alloc for one example.

Heap hardening/exploit mitigations are hard to get right. Ptmalloc has done a decent job save for 1 or 2 techniques lost to time. As a result modern exploits target application specific data (C++ object vtables, function pointers, etc) and not heap meta data.

[1] I wrote this blog post and I audit code for a living

Ok, I stated "many", but not all or most.

I of course have not your expertise. I am also not totally up to date with the newest mitigation techniques. I think, it depends largely on the focus of the allocator. Most have only speed on focus and that is what is implemented.

> So the focus on the self-written allocator seems to get a little biased in this discussion.

I feel like I'm missing something here. Where are the self written allocators? This article states that the OPENSSL_Malloc is simply malloc by default.

I did not check the contents of the article on correctness. It seems to me, that OPENSSL_Malloc is really only malloc by default -- but I lack the time to really check.

I wrote this sentence, because there is a big discussion ungoing, with some people blaming the self written allocator of OpenSSL. See the link, I gave. My point is, that even when it is so, having the standard allocator does not guarantee that you find this bug /in testing/ -- what is claimed in that article.

Correction: When you read the article of this topic exactly, the freelist implementation (allocator) is used for recvd data. That is what started the discussion.

There is a big discussion going on that could be misleading.

To clarify, are we calling the freelist implementation, which the heartbeat code does not use, an allocator?

Update: I agree with the author that it is wrong to point the blame at the freelist implementation. If every C application that manages the reuse of commonly used data structures is doing wrong, then pretty much every modern server application will have to be re-written -- for instance Apache [1].

C is fast and portable and binds to just about any language which is why OPENSSL is in such wide use. Maybe it would be better to use more 'secure' languages like Go, but if OPENSSL was written in Go, how many applications would use it? I'd say almost none.

[1] https://apr.apache.org/docs/apr/1.5/group__apr__pools.html

A free list implementation is some kind of self made allocator.

I believe there is some confusion (at least I was confused until I looked at the code) that the library had universally replaced malloc with another allocator. While the library allows users to provide their own allocator, it is not done by default.

The code in question does not use the freelist implementation. It goes directly to OPENSSL_Malloc which is basically malloc.

What happened was the exploit allowed remote clients to read beyond the size of the buffer allocated by OPENSSL_Malloc. There just happened to be data from the freelist sitting next to it.

Even if the freelist implementation wasn't used, that wouldn't have prevented this exploit. There would just be something else sitting there in memory, but we can't predict what that would have been.

As much I understood the article (had also to look twice), the problem is not the OPENSSL_Malloc near the troublesome coding, but an other allocation for the received data. And that is (as much I understood) handled by the freelist. But the portions where of course allocated by normal mallocs.

OpenSSL has an abstraction layer over malloc, which uses an internal stack-based free-list, and which it uses for all allocations. It's not an actual malloc(3) implementation, which is half the problem.

If OpenSSL had just written a malloc(3) implementation that sat in a separate shared-object--the way that, for example, jemalloc does--then switching it out would be a simple and obvious linker argument. Any developer who had used a project that relies on an alternate malloc (e.g. redis) would know that "test while linked against system malloc" is a crucial step.

Instead, you have to learn that there's a specific OpenSSL debugging flag that disables the "caching behavior" of the malloc wrapper, causing it to become a passthrough to regular malloc. None of the tests use this flag.

Here's where the freelist operations are defined: https://github.com/openssl/openssl/blob/21e0c1d23afff48601eb...

But where are they called from?

Update: To answer my own question the freelist is used in

    ssl3_setup_read_buffer()
    ssl3_release_read_buffer()
    ssl3_setup_writer_buffer()
    ssl3_release_write_buffer()

This is not the same as universally replacing the allocator.

> OpenSSL has an abstraction layer over malloc, which uses an internal stack-based free-list, and which it uses for all allocations

To be clear, this is NOT TRUE. The freelist is used for some allocations, but not in the patch in point. Here is the patch that addresses the bug: https://github.com/openssl/openssl/commit/731f431497f463f3a2...

It clearly uses OPENSSL_Malloc which is basically the equivalent of calling malloc.

https://github.com/openssl/openssl/commit/731f431497f463f3a2...

On some platforms, malloc(3) and free(3) are implemented in terms of mmap(2) and munmap(2). Which means, if you go beyond your buffer space, you'll end up in an unmapped page and cause a SIGSEGV or end up in NULL data.

Since they used the freelist/malloc wrappers, they avoided this trap.

Plus, IMO, if you're implementing a freelist, you're implementing 80% of what a malloc is anyways. It's _mostly_ freelist handling, the only way to add memory to a process aside from mmap(2) is setbrk(2). There's no way to give memory back, so they only thing left for a malloc implementation to do is create a freelist to reissue memory previously freed.

You can say you're still backed by malloc(3) if you want, but you're only getting the setbrk(2) handling and throwing away everything else it's trying to provide you.

If you read my statement correctly, I did not say, that some forms of malloc would not be helpful in finding or even preventing the bug. I only attacked the statement, that some kind of malloc could magically help finding a bug in testing -- a bug that only appears when provoked (and when you come up with the idea, provoking it, you don't need any malloc anymore to find it).

Nobody said that it would help find it in testing. You're attacking a strawman.

Edit: although that is also false if you are using a quickcheck-like testing framework.

Edit2: the article may have been claiming that, but Ted did not claim that, nor did Theo.

I don't understand what you want to tell me with all you said, after the "Plus".

The point of self implemented allocators is not, that they are all better than malloc or that the implementor is more wise or that there is some magical additional stuff or even that some code is saved (the oposite is true) ...

The point is only, that malloc implements the most general case (what it must, since it is the magical "I can do it all" tool (at least in memory management)). When you go away from the general case you can find several specific cases, that can be implemented specialized for this case -- and those specific cases could be implemented very much more efficient (concerning speed, not coding!).

That is the whole point. Also there are some specific allocators that also implement some safeguides for common memory problems. But that seems to be more of the exception, when I take into account the feedback.

What I was trying to convey is that if you say "I'll just implement a better freelist" you're really just fooling yourself. What you are doing is re-implementing most of malloc on top of malloc, poorly.

If would have been better if they had gone whole-hog with a malloc library and built it from the ground up for their specific purposes instead of half-assed "freelist speedhacks."

He is right, after all, it is just a bug, like so many others some of them we are not aware off, and it did not break internet. It just may leak information, if and only if you really know your job in coding and are very good. People that could be hired by organization with the money and the need for it? Nooo.. You have to be paranoid to think so. And ... then throwing the stone at openssl would be like distrusting all softwares: they all have bugs. By the way, I never saw a software with a legal liability clause for bus for a software... so ... if a user have a problem or money loss caused by softwares, then he is the responsible for his choices.

There was some discussion here[1] that OpenSSL had implemented their own allocators, but doesn't seem to be the case. As this article points out, a quick review of the code will show that OPENSSL_Malloc() is simply malloc by default[2].

[1] https://news.ycombinator.com/item?id=7565064

[2] https://github.com/openssl/openssl/blob/a898936218bc279b5d7c...

This analysis doesn't support the conclusion that the bug isn't severe at all. And telling the reader that nobody cares about their email is rather smug. The egos in the security community are out of control.

There have been remote code execution vulnerabilities in OpenSSL in the past, yet no one panicked. Those vulnerabilities gave you what heartbleed does and more. The worst part about heartbleed is that its easy to exploit and the tools to do it are readily available. Its bad, as I state, but not the end of the world. The main point of the post was to correct the public analysis of the freelist implementation.

Please don't mistake a poor attempt at humor for ego.

I don't see a byline here; is this written by Chris?

Yes, it was.

Reply on news.ycombinator.com