Hacker News Reader

Comment by cesarb | Hacker News Reader

I wonder how long until programs start to accidentally depend on it, the same way some programs currently accidentally depend on freeing memory not immediately overwriting it.

For instance, I can imagine a program which accidentally follows a dangling pointer to an already freed structure, and reads another pointer from within that structure, not crashing because it ends up checking this later pointer against NULL... until the stars align and the memory used by the freed structure has been returned to the operating system, or overwritten by a later allocation.

View on HN

I urge anybody using a C family language to become familiar with AddressSanitizer [1], a tool that (with support from the compiler) detects an impressive variety of memory bugs at runtime. The scenario you describe would result in the program immediately crashing with a read-after-free error, along with stack traces for the most recent calls to malloc and free that involved that address. It's remarkably useful and straightforward, and Apple's tooling has good support for it, along with Xcode integration [2].

[1] https://clang.llvm.org/docs/AddressSanitizer.html

[2] https://developer.apple.com/documentation/xcode/diagnosing-m...

This. I personally think ASAN and it's friends are some of, if not the, best things to happen to C/C++ in the last decade. We use it at work, I use it in all my personal projects. We've found "countless" bugs with ASAN together with LSAN.

It should be part any project from the start, run as much as possible with ASAN when the performance hit can be afforded.

Just confirming, but you're taking about Leak Sanitizer? https://clang.llvm.org/docs/LeakSanitizer.html. Not really familiar with these tools, definitely not the acronyms.

Sorry, yes, I meant Leak Sanitizer.

No need to be sorry! I appreciate it!

It was actually in the link the parent posted. I just missed it at first.

Agreed! Likewise, if you're doing anything multithreaded, ThreadSanitizer (TSAN) is a must!

I’m using sanitizers on macOS for few years now. Doing cross-platform, ~~it really feels inferior~~ I really miss them on MSVC side. (wrt to exclusive asan since VS2019)

Still, in some cases you have some tiny uninitialized member for example that is much harder to find. So with this malloc it might even cover those more.

On Mac, malloc has several options to help find such bugs. If you set MallocScribble=1 in the environment, it fills blocks with 0xaa on alloc and 0x55 on free.

On Linux, running `env MALLOC_PERTURB_=43605 myprogram` will do the same.

That should be the default behaviour, perhaps with random values too.

It’s easier when it is a specified value. Easier to see in a debugger and to diagnose. Sure, it can occasionally be interpreted as meaning something, but so can random gibberish. As long as everything is overwritten, there is no privacy issue.

> Sure, it can occasionally be interpreted as meaning something, but so can random gibberish.

Some gibberish is better than others. For instance, on x86 (64-bit, 32-bit, or even 16-bit), filling with the 0xcc byte means that, if executed as code, it will trap (0xcc is the single-byte INT3 "single-step breakpoint" instruction). And for most Unix-style operating systems, which put the kernel in the high half of the address space, the filler bytes should have the high bit set, so that treating them as pointers will trap (the pointer will have its high bit set, pointing to kernel memory, which user space is forbidden from accessing).

Or just use valgrind.

Or just use a memory safer programming langage.

If your use case fits into capabilities of such language, and you have the necessary libraries, and bindings, etc., then why not?

Having a different pre-defined value for allocation vs deallocation scribble is nice because you can then tell if the error happened pre-initialization or post-free. $AA and $55 also weren't chosen by accident one assumes, $AA is 1010_1010 and $55 is 0101_0101. Consistent values help with reproduction.

The Linux setup where you can choose your own scribbles seems to meet your needs no?

> For instance, on x86 (64-bit, 32-bit, or even 16-bit), filling with the 0xcc byte means that, if executed as code, it will trap (0xcc is the single-byte INT3 "single-step breakpoint" instruction).

Trying to execute off the heap is going to trap anyways.

It’s slower.

For the confused, 43605 in decimal is 0xaa55 in hexadecimal.

As long as Apple's allocator continues to zero freed memory, programs accidentally depending on that behavior will only be a problem for other operating systems.

> As long as Apple's allocator continues to zero freed memory, programs accidentally depending on that behavior will only be a problem for other operating systems.

As I mentioned, only until that freed block is reused for a new allocation, which means it will no longer be all-zeros; or until the memory allocator decides it's time to unmap that region, which means it will trap (SIGSEGV or similar).

> which means it will trap

Is it really a problem? Failing harder and earlier is a good thing if it means the bug has more chances of being detected and fixed.

Memory allocators generally don't unmap anything, because the performance hit on any other threads running is severe.

When memory is unmapped, caches of memory mappings of other threads are discarded, via inter-core interrupts. Then they get misses until their cache is restored.

In what might be a multi-thread program, you don't fool with the memory map without very good reasons. Mapping new pages, or marking a page r/o or r/w is OK; anything else, probably not.

> When memory is unmapped, caches of memory mappings of other threads are discarded, via inter-core interrupts.

ARM architecture does not need to interrupt for TLB shootdown.

Possibly but there's the potential reverse problems where Apple programs start depending on newly allocated memory being zero .... and those programs start failing elsewhere ...

Might fail there, too, but only sometimes, if the memory gets allocated again.

Well the change is to zero out memory on deallocation. So on allocation it may or may not be zero depending on whether it was previously freed, right?

not necessarily, even if was freed there might be defragged free blocks that still contain heap metadata by the time it gets reallocated

Or until the particular allocation moves to a custom/separate heap or other non-system-malloc allocator. That sort of tuning is very common in big apps.

This is better than the status quo of use-after-free being abused for security exploits as part of an exploit chain for sandbox escape.

  "the same way some programs currently accidentally depend on freeing memory not immediately overwriting it."

Maybe a dumb question, but what do you mean by this? Could you give a C/C++ pseudo-example?

Not the poster, but it’s fairly easy to accidentally do the equivalent of

   free(f);
   f->close();

in code that does a few malloc, if/else’s, etc. between those calls. Stuff like that gets hard to spot in more complex code, especially in the presence of C++ copy constructors, move constructors, etc, where the compiler helpfully inserts that call to free for you.

Bugs in copy constructors also can lead to double frees when the destructors of an object and its copy both call delete on a pointer that they think they exclusively own. Between those two destructor calls, you can easily get a use after free. Example: https://stackoverflow.com/a/64014035

If the allocator doesn’t change the content of freed blocks, such bugs can go unnoticed, making the program “accidentally depend on freeing memory not immediately overwriting it”.

> especially in the presence of C++ copy constructors, move constructors, etc,

You only mean the former, not both. The only reason for the existence of the giant rats' nest of complexity that is C++ move semantics is eliminating this particular mistake.

Ooh, I have a good one for this. If you have a C++ `std::string` object passing into a C API that takes a null-terminated `char *`, there's a convenience method `std::string::c_str()` to produce the C string from the C++ one.

    char const* inputFilename = std::string("path/to/file").c_str();
    call_to_C_API(inputFilename);

Boom. You can typically access `inputFilename` until the next allocation on the stack, but it will almost always go bad quickly. This isn't allocation on the heap, which usually requires another level of indirection, but you get the idea.

> This isn't allocation on the heap, which usually requires another level of indirection, but you get the idea.

Isn't it allocating in the stack due to the "small string optimization"? Making string contents longer (or using a C++ standard library which doesn't have a "small string optimization" in the first place) would be enough to have it be allocated in the heap instead.

No, with longer string contents the std::string will internally allocate and store on the heap, but as the containing std::string is a temporary it will be pop'd of the stack (stack pointer moved, which is why it works "sometimes") and its destructor called at the end of the statement (at the semicolon), freeing the heap allocated memory (which, as parents are discussing, may still work because nothing has yet overwritten the recently freed memory).

Changing it to:

    call_to_C_API(std::string("path/to/file").c_str());

Makes it valid because the std::string object will live until the semicolon.

Reply on news.ycombinator.com