Hacker News Reader

> IMO the right way to have private members in C structs is... to document that members shouldn't be touched directly, perhaps using a special naming convention or embedding the publicly-accessible members in a dedicated sub-struct to prevent confusion.

Reminds me of that one time when glibc broke the whole of Debian for s390 architecture by changing the fields in the jmp_buf struct (which is public): [0].

[0] https://lwn.net/Articles/605607/

To achieve a reasonable level of encapsulation in C, a header file must be seen as a public-only interface. It should declare only the structs that are relevant for the user of the module. If that's "struct my_module_handle { ... }", declare it and document the corresponding accessor and modifier functions. Everything else must reside in the C source file with internal linkage (static storage class). The whole source file is your implementation.

There is an anti-pattern where header files are used for all the declarations needed internally by the source file. Including (pasting verbatim with the preprocessor) that file from another module would bring in all the unnecessary declarations.

I think what you say makes complete sense at module-level (as in, for a standalone lib for instance) but I never bother segregating things internally within a lib/module/exe and rely on good documentation and coding practices to avoid having member mutations all over the place.

If I code in Rust or C++ I can use namespacing and public/private to give every single object in the codebase a clean interface, but in C doing that is just frustrating, not to mention potentially inefficient.

Opaque pointers usually impose the restriction on the API such that in order to use the handle one has to dynamically allocate the object on heap. That's a quite unfortunate tradeoff IMO.

Why not have the API take an allocator as a parameter? Pass in pointers to malloc, realloc, and free. Then the library can use your static allocator (or some 3rd party malloc such as jemalloc, or your own arena allocator for that matter) or it can default to the system one if you pass null pointers instead.

Callback based APIs are better, but not good enough. The problem remains that the user still has to deal with some potentially undesirable constraints:

* I may be running in an environment where allocating memory is an asynchronous operation. A callback based API forces me to block, which can cause unpleasant side effects like halting an event loop.

* In my experience, some libraries with custom allocation hooks forget to define one or both of the two basics in the callback signature: a "context" or "user data" parameter, and a way to return an error.

The proper solution is to decouple memory allocation from object initialization entirely. There are two different approaches to this:

1. Expose get_foo_size() and get_foo_align() functions that return the size and alignment that the opaque foo struct needs (at runtime, of course). Then I as the user can allocate that memory, and initialize my opaque foo objects in-place:

  size_t foo_size = get_foo_size();
  void* buf = alloc_aligned_memory(foo_size * 1000, get_foo_align());
  for(size_t i = 0; i < 1000; ++i) {
    int err = foo_init(buf + i * foo_size, /* params here */);
  }

2. Define foo_init(void* buf, size_t len, ...) which attempts to initialize an opaque foo object in the buffer defined by [buf, buf+len). If the buffer does not have enough space, return an error. Otherwise, return the number of bytes actually used by the object.

That second method works fine if you just want a buffer of a bunch of foo’s and they all happen to be the same size. Not so fine if foo is a data structure in its own right with growable capacity.

That is possible and reason why I said "usually". Still, it's an unfortunate complication because you need to manage the pool of your (static) objects now. Also, it's not possible to use stack with this scheme.

there are ways around this, if VLAs are allowed.

  // in <opaque_foo.h> 
  typedef struct opaque_foo opaque_foo_t;
  size_t opaque_foo_sz(void); 
  void opaque_foo_init(opaque_foo_t* foo) 

  // in your code, which you could write a helper macro for if you were so inclined
  char opaque_foo_mem[opaque_foo_sz()]; 
  opaque_foo_t * my_foo = (opaque_foo_t*) opaque_foo_mem;
  opaque_foo_init(my_foo);

This approach violates alignment requirements causing misaligned accesses that on some architectures is mitigated by generating extra instructions while on some other architectures it's a violation (e.g. segfault).

There is no way for a compiler to infer the alignment requirement of a struct because it does not see its definition. You would always have to align the char buffer by hand but you cannot do it for the same reason compiler cannot.

What you can do is to always greedily align the char buffer to the strictest (largest) fundamental requirement for that platform - in other words alignas(max_align_t).

And this is actually what alloca() does for you by default. From https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html

> The object is aligned on the default stack alignment boundary for the target determined by the BIGGEST_ALIGNMENT_macro.

Doesn't this violate strict aliasing?

yes, though you can fix that with compiler flags (or, #pragma if you want strict aliasing elsewhere in your code)

alternatively, gcc supports VLAs in unions, but I don't think clang does, but that makes it extra annoying to do.

edit: apparently you can probably apply the may_alias attribute to the type? Or you could try using transparent_union. No idea if clang supports either...

At that point, it is IMO better to obtain that block from alloca()[1]—it’s not standard, but where it’s available the compiler will treat the result as untyped for the purposes of aliasing. (If you’re on GCC/LLVM, __builtin_alloca_with_align is also an option, although note that the memory it returns may not outlive the current block—similar to a standard automatic variable, but unlike memory from traditional alloca.) ISO C has a gigantic hole when it comes to obtaining and recycling untyped memory, pretending the hole is not there isn’t going to help.

[1] https://nullprogram.com/blog/2019/10/28/, discussed at the time at https://news.ycombinator.com/item?id=21374863

And alignment?

Just use alloca.

  // in your code, which you could write a helper macro for if you were so inclined

  opaque_foo_t *my_foo = alloca(opaque_foo_sz());
  opaque_foo_init(my_foo);

yes, you may need an alignas depending on platform (though you probably want it even if unaligned access is supported).

char* can alias anything

The strict aliasing optimization in gcc in theory can cause problems (which is why many, including the Linux kernel, disable that)

This is how it's supposed to be done but you always end-up moving them to header just to make writing unit tests less painful.

This is a smell. Your unit tests should not have to rely on internal implementation details.

And in the worst case you can have module-private headers. No need to pollute your interface.

Are unit tests common in C? In the mid-2000's, I worked on an "enterprise" system, written in C and C++. There were about 300,000 lines of code, maybe 10 tests. This thing was the core of a billion dollar business

That's very broken; more usual practice at the time was to have large numbers of tests, aiming for good code coverage. I've sometimes seen too many full-tool tests and few true unit tests (tests that just test functions for correctness), but "maybe 10 tests" is frightening.

This particular company was very heavy on manual testing, unfortunately. (This was roughly 2003 - 2006 ish.)

They don't have to and shouldn't, but it's convenient. That's the parent's point ("to make writing unit tests less painful").

I like the way that Windows does it, where they have as the first element of the struct a double-word size (dwSize) element that records in 32-bits the size of the structure. The size essentially acts as a version identifier, as long as you never rearrange the fields and only append fields for new versions. The opaque functions test the value of the dwSize element to see what actions can be performed on the object.

The code that you develop can still access the member fields directly, and those accesses can be inlined and optimized aggressively by the compiler.

This implies a branch statement in every function call, doesn’t it?

There's zero cost abstractions and then there's zero features abstractions.

The cost of that branch is insignificant next to that of the syscall you opted to make.

Link-time optimization means that you're probably not going to take that much of a performance hit.

But yes, opaque structs do enforce that it will be treated as a plain pointer, and the compiler (usually) cannot treat it as an aggregate of variables.

If a linker did that, changing the layout would be an ABI-breaking change. I think this opaque struct design is most common for dynamically loaded libraries, where link time optimisation does not occur (unless dynamic linkers got a lot more fancy recently).

The Java analogue of opaque structures is factory methods. Their only purpose is to hide the new keyword. That's necessary precisely because new introduces a hard dependency on the constructed type at the binary interface level, it gets literally emitted into the byte code. The supertypes returned by those methods are just there to serve as compatible pointers to the real types.

> is not exactly like private

That seems like a straw man to me. He never said it was exactly equivalent - he said it provided encapsulation and isolation. 'private' is another language's mechanism which does that but they're obviously not identical.

> The code remains binary compatible even, no need for a recompile if you're shipping a shared lib. This is something just using private members doesn't achieve since with private members the compiler still knows and uses the layout of the struct, it just forbids access to it.

There is somewhat common PIMPL idiom to work around the binary compat issue. Iirc there were some macros floating around to make it easier to manage.

What do the macros do? Isn't this as easy as forward declaring the impl class, adding std::unique_ptr<Impl> as a private field and have public methods refer to the field? I'm struggling to understand why macros would help here.

Perhaps the macros help you forward methods on the outer class to methods on the impl class? While your approach of having public methods refer to the field also works, it’s nice to have public and private methods in the same place (the impl class’s definition) and using the same syntax (neither having to go through the impl field).

> it’s nice to have public and private methods in the same place (the impl class’s definition)

In my experience, the impl class is usually defined in the main class's cpp file.

I'm also only familiar with this idiom in C++, but based on the description in the parent comment, I suspect that this is sometimes used in C too, in which case you obviously can't use unique_ptr or private fields; maybe macros might be a way to avoid having to write a bunch of boilerplate to achieve a similar effect?

> That's why you can even find C++ libraries use this idiom even though C++ obviously has `private`. It's about having a stable, opaque API.

In C++ this is a popular idiom and a standard technique covered by multiple references under the name pointer to implementation (pimpl).

This is not a C thing beyond the point that C has pointers.

> On the other hand because of this added indirection, there's usually a greater performance hit to accessing these opaque structs since code can't be inlined.

Is this still true? Link-time optimization, which includes inlining, seems to be all the rage these days.

For static linking quite possibly, although you could still have un-optimizeable sources of overhead because of having to indirect through pointers everywhere, for instance in nested structs. Something like a->b->c->d is going to be more expensive than a.b.c.d, for instance.

I'm also not sure that dynamic linkers are typically smart enough to do this type of optimization, but I admit that I'm not familiar with the state of the art in dynamic linking technology.