Hacker News Reader

Comment by eska | Hacker News Reader

In practice this is mostly avoidable. There are many libraries that do not allocate at all by forcing the caller to provide the memory. The library may tell in advance how much memory is necessary, or report in hindsight whether it was enough for the operation. This leads to a style of allocating ahead of time and also considering worst case memory size.

Otherwise I prefer libraries that allow setting the allocation function at least.

View on HN

The especially great thing about this approach is it lets you put things on the stack when appropriate too. It's really annoying when some library forces you to spam the heap with short lived, easily scoped objects.

True. Also great for thread safety. And avoiding shared access performance pitfalls

This is the correct approach. Let the caller allocate the memory, and if necessary provide a function to calculate the required memory beforehand, and for the caller to specify the size of the memory passed to the primary function. For dynamically-sized results, another option is to provide iteration functions (like when walking a directory tree) in order to simplify memory sizing for each individual function calls.

Yes, it is educated that way. However, for the case requires some scratch space, dynamic allocations can be easier to manage, less error prune and with less overall memory usage (you don't need to preallocate worst case amount).

That's why even people knew, many C APIs still return dynamic allocated objects or simply let you inject malloc / free if you want more control.

This is a roundabout way to say: if you aspired to provide APIs with zero dynamic allocation, go ahead. But if you find yourself struggling with more complicated code as a result, think about just letting a little bit dynamic allocations may help.

There is a funny version of this:

Caller provides memory and function fails if there is to little memory, but writes the the required (variable) amount of memory to an `out length` pointer.

Effectively leading to a common pattern of:

1. call function with empty buffer (or buffer or arbitrary size)

2. allocate buffer

3. call function again with properly sized buffer

4. add a loop if between 1 & 3 the required buffer size can change

its fascinating how well it works for some of it's common use cases and how subtle but badly broken it can be for other use cases :=)

That's classic "Microsoft style" and I've always hated APIs like that because they add unnecessary complexity to the code of all their callers.

I once had the pleasure of using the Microsoft C API for creating an HMAC:

https://docs.microsoft.com/en-us/windows/win32/seccrypto/exa...

They managed to mix both patterns into a ghastly 9 function call!

The example above speaks for itself but I kind of see the "we provide you with memory you need to call 'FreeX' on" and the "call the function twice, once to find out how much memory you need to allocate" patterns to be equally annoying to use.

It’s nice if you would prefer to use a custom allocator, or put small buffers on the stack. But yeah, it is more annoying to use.

You can’t just dump ‘small’ buffers on the stack in security sensitive code; you never know how big ‘small’ can get and you have to define what happens if it’s too big anyway.

Yeah, but that is what allows Windows to keep its ABI, while extending some struct across Windows versions.

Callers allocate the right size no matter what OS version, and can only see the fields exposed on the version they were built for.

Of course this very paradigm, combined with the fact that C doesn’t have a proper pointer-and-length type, led to `gets` and around n million other security disasters when bad APIs would just take a pointer without a length and assume there’s "enough" space…

Anyway, the whole "caller allocates" concept doesn’t work too well in the particular case of `gethostbyname`, which as the author mentions, is a complex struct containing several pointers and double pointers, with a potentially unlimited number of different-length allocations one would have to make!

For things that create complex in-memory data structures, an alternative is to use callbacks instead. Users can still create such data structures if they want to but at least that way heap allocations are optional.

Caller allocates works fine. gethostbyname would effectively serialize all of the result data into the supplied buffer. All of the pointers in the struct would be internal to the buffer (they point to the serialized data) so they don't need to be freed.

Yeah, but it would be nigh impossible to guess in advance how much space would be needed. The "solution" where the caller tries again with exponentially increasing buffer sizes really doesn't fit my concept of a good API. And if the callee calculates and returns how much memory it would need, then the caller needs to heap allocate to make use of that information (or resort to nonstandard functions like `alloca`). Honestly best just to return a pointer to a dealloc function (which could simply be `free` if the struct and the data it points to are indeed allocated as a contiguous block of memory).

Please excuse my little experience with this topic, but when the article says this:

> If this structure is dynamically allocated by gethostbyname() and returned to the caller, either you need an additional API function to free it or you have to commit to what fields in the structure have to be freed separately, and how

With your approach, wouldn't you need to free all the fields in the structure separately as well? Because, what's the difference between the library allocating the memory -with the problems the article points out- and allocating it yourself?

Reply on news.ycombinator.com