Hacker News Reader

On some platforms, malloc(3) and free(3) are implemented in terms of mmap(2) and munmap(2). Which means, if you go beyond your buffer space, you'll end up in an unmapped page and cause a SIGSEGV or end up in NULL data.

Since they used the freelist/malloc wrappers, they avoided this trap.

Plus, IMO, if you're implementing a freelist, you're implementing 80% of what a malloc is anyways. It's _mostly_ freelist handling, the only way to add memory to a process aside from mmap(2) is setbrk(2). There's no way to give memory back, so they only thing left for a malloc implementation to do is create a freelist to reissue memory previously freed.

You can say you're still backed by malloc(3) if you want, but you're only getting the setbrk(2) handling and throwing away everything else it's trying to provide you.

If you read my statement correctly, I did not say, that some forms of malloc would not be helpful in finding or even preventing the bug. I only attacked the statement, that some kind of malloc could magically help finding a bug in testing -- a bug that only appears when provoked (and when you come up with the idea, provoking it, you don't need any malloc anymore to find it).

Nobody said that it would help find it in testing. You're attacking a strawman.

Edit: although that is also false if you are using a quickcheck-like testing framework.

Edit2: the article may have been claiming that, but Ted did not claim that, nor did Theo.

I don't understand what you want to tell me with all you said, after the "Plus".

The point of self implemented allocators is not, that they are all better than malloc or that the implementor is more wise or that there is some magical additional stuff or even that some code is saved (the oposite is true) ...

The point is only, that malloc implements the most general case (what it must, since it is the magical "I can do it all" tool (at least in memory management)). When you go away from the general case you can find several specific cases, that can be implemented specialized for this case -- and those specific cases could be implemented very much more efficient (concerning speed, not coding!).

That is the whole point. Also there are some specific allocators that also implement some safeguides for common memory problems. But that seems to be more of the exception, when I take into account the feedback.

What I was trying to convey is that if you say "I'll just implement a better freelist" you're really just fooling yourself. What you are doing is re-implementing most of malloc on top of malloc, poorly.

If would have been better if they had gone whole-hog with a malloc library and built it from the ground up for their specific purposes instead of half-assed "freelist speedhacks."

Agree with most of this except for "and many specialized allocators have their own security mitigation build in." This is generally not true in my experience [1] when it comes to unlink checks, guard pages and the like. Often a generic homegrown allocator will call mmap/VirtualAlloc and chunk the page(s) up into a doubly linked list or simple arena allocator.

Sometimes a default generic allocator will have design properties about it that make exploiting certain bug classes difficult (e.g. WebKits RenderArena and Use-After-Free). There are exceptions of course, see Chromes Partition Alloc for one example.

Heap hardening/exploit mitigations are hard to get right. Ptmalloc has done a decent job save for 1 or 2 techniques lost to time. As a result modern exploits target application specific data (C++ object vtables, function pointers, etc) and not heap meta data.

[1] I wrote this blog post and I audit code for a living

Ok, I stated "many", but not all or most.

I of course have not your expertise. I am also not totally up to date with the newest mitigation techniques. I think, it depends largely on the focus of the allocator. Most have only speed on focus and that is what is implemented.

> So the focus on the self-written allocator seems to get a little biased in this discussion.

I feel like I'm missing something here. Where are the self written allocators? This article states that the OPENSSL_Malloc is simply malloc by default.

I did not check the contents of the article on correctness. It seems to me, that OPENSSL_Malloc is really only malloc by default -- but I lack the time to really check.

I wrote this sentence, because there is a big discussion ungoing, with some people blaming the self written allocator of OpenSSL. See the link, I gave. My point is, that even when it is so, having the standard allocator does not guarantee that you find this bug /in testing/ -- what is claimed in that article.

Correction: When you read the article of this topic exactly, the freelist implementation (allocator) is used for recvd data. That is what started the discussion.

There is a big discussion going on that could be misleading.

To clarify, are we calling the freelist implementation, which the heartbeat code does not use, an allocator?

Update: I agree with the author that it is wrong to point the blame at the freelist implementation. If every C application that manages the reuse of commonly used data structures is doing wrong, then pretty much every modern server application will have to be re-written -- for instance Apache [1].

C is fast and portable and binds to just about any language which is why OPENSSL is in such wide use. Maybe it would be better to use more 'secure' languages like Go, but if OPENSSL was written in Go, how many applications would use it? I'd say almost none.

[1] https://apr.apache.org/docs/apr/1.5/group__apr__pools.html

A free list implementation is some kind of self made allocator.

I believe there is some confusion (at least I was confused until I looked at the code) that the library had universally replaced malloc with another allocator. While the library allows users to provide their own allocator, it is not done by default.

The code in question does not use the freelist implementation. It goes directly to OPENSSL_Malloc which is basically malloc.

What happened was the exploit allowed remote clients to read beyond the size of the buffer allocated by OPENSSL_Malloc. There just happened to be data from the freelist sitting next to it.

Even if the freelist implementation wasn't used, that wouldn't have prevented this exploit. There would just be something else sitting there in memory, but we can't predict what that would have been.

As much I understood the article (had also to look twice), the problem is not the OPENSSL_Malloc near the troublesome coding, but an other allocation for the received data. And that is (as much I understood) handled by the freelist. But the portions where of course allocated by normal mallocs.

OpenSSL has an abstraction layer over malloc, which uses an internal stack-based free-list, and which it uses for all allocations. It's not an actual malloc(3) implementation, which is half the problem.

If OpenSSL had just written a malloc(3) implementation that sat in a separate shared-object--the way that, for example, jemalloc does--then switching it out would be a simple and obvious linker argument. Any developer who had used a project that relies on an alternate malloc (e.g. redis) would know that "test while linked against system malloc" is a crucial step.

Instead, you have to learn that there's a specific OpenSSL debugging flag that disables the "caching behavior" of the malloc wrapper, causing it to become a passthrough to regular malloc. None of the tests use this flag.

Here's where the freelist operations are defined: https://github.com/openssl/openssl/blob/21e0c1d23afff48601eb...

But where are they called from?

Update: To answer my own question the freelist is used in

    ssl3_setup_read_buffer()
    ssl3_release_read_buffer()
    ssl3_setup_writer_buffer()
    ssl3_release_write_buffer()

This is not the same as universally replacing the allocator.

> OpenSSL has an abstraction layer over malloc, which uses an internal stack-based free-list, and which it uses for all allocations

To be clear, this is NOT TRUE. The freelist is used for some allocations, but not in the patch in point. Here is the patch that addresses the bug: https://github.com/openssl/openssl/commit/731f431497f463f3a2...

It clearly uses OPENSSL_Malloc which is basically the equivalent of calling malloc.

https://github.com/openssl/openssl/commit/731f431497f463f3a2...