The Unix C library API can only be reliably used from C | Hacker News Reader

The Unix C library API can only be reliably used from C

utcc.utoronto.ca

91 pointsjandeboevrie70 comments

This is a fairly common problem when trying to interface other languages with any C API, not just errno. The FFI-library of your language can usually call straight into C-functions but then it turns out half of the C API is implemented in macros which can't be called nor converted automatically, so you have to be re-implement them all in the new language.

Funny because usually the reason for having a C API is to provide universal compatibility to be called from any language. If you are doing this, please avoid macros.

I ran into that at work a few years back. I was tasked with integrating OpenSSL into the commercial Smalltalk used for the project (last updated in 1999, for more or less good reasons). A priori, not an unreasonable task as the Smalltalk had a pretty good FFI mechanism. Except of course the neverending cavalcade of functions documented in the OpenSSL docs that were actually preprocessor macros. In the end I wrote a Python script that parsed the C headers and generated FFI entry points for normal functions and Smalltalk code for preprocessor macros that weren't simple literals. The result was a small mountain of code, but it made all of OpenSSL available to our application, which was great.

You’re still lucky it wasn’t like DirectDraw or something WinAPI in general. I had to reinvent entire preprocessors and investigate implicit compiler defaults to be able to wrap that, and still there was LOWORD, MAKE_XYZ, etc, which were official arbitrary code-in-a-header interfaces. Preprocessor is a scourge of an API.

Hence why since Vista all major new Windows APIs are COM based (now UWP).

COM is a (distributed) object model. It's mostly about structured object lifetimes. I don't think COM offers any solution for replacing bad use of preprocessor in APIs where we couldn't simply use plain C functions instead.

> COM is a (distributed) object model.

Not at all, that is why DCOM exists.

COM offers a language agnostic component model for interoperability between development stacks on Windows, and a modern OS ABI.

I meant to say inter-process.

COM is also usable in-process.

It is, that's why I wrote it in parentheses. And all of that wasn't my point anyway.

> a modern OS ABI.

Last I looked at it, it was created in 1993 (enterprise OOP craze?) and was just regular DLLs containing regular code, wrapped in tons of symbols cruft to make a "dynamic classes protocol" work.

If you actually have to deal with it directly from a non-managed language, I'm very doubtful that it's nicer than a well-written C header file. What I did when I saw MFC macros and 40 lines of boilerplate to marshal arguments to call into a single entry point, was to RUN, which is probably a good idea unless you want to put tons of supporting tooling and code generation in place.

Sorry to say, then you are stuck in the past and have a C tainted view of COM infrasture.

Using COM from C++ Builder, VB 6, VB.Script, Delphi, Perl or Python, to use 90's only examples, did not have anything to do with "MFC macros and 40 lines of boilerplate to marshal arguments to call into a single entry point".

No one uses MFC to create COM nowadays, unless they are dealing with mid-90's code.

It is modern, because a component based ABI, with language interoperability in mind, is much better than a fossiled ABI designed for PDP-11 clones.

It may well be that I missed an important development, but the underlying cruft of course didn't change. It's DLL based object code, just like regular C code, except that a lot of boilerplate is required to be a "blessed COM" DLL. https://en.wikipedia.org/wiki/Component_Object_Model#Technic...

> No one uses MFC to create COM nowadays, unless they are dealing with mid-90's code.

I guess that's what I was given...

Now tell me what will be easier to call from my hobby programming language, COM or regular object code?

> because a component based ABI

> It is modern, because a component based ABI, with language interoperability in mind, is much better than a fossiled ABI designed for PDP-11 clones.

It is literally the same "fossiled ABI", wrapped in more shitrolls to hold all the crap.

If you just mean the APIs are more pleasant to use, then maybe that is so, due to the structured objects system, if you have all the tooling in place. I'm not sure I've ever used it from a managed language, and in general, hiding cruft by adding more cruft is not my cup of tea.

Here is one example that I recently found. It shows the bullshit you have to go through to interface with COM as a C programmer: https://pastebin.com/3YvWQa5c

The boilerplate is what happens when some people insist in using C instead of C++ or other modern languages to use COM, so they need to write manually what the compiler would do for them in more modern alternatives.

Note that the example I posted is written in C++, by Jonathan Blow (you do not need to like him but I hope we can agree that he is an accomplished programmer). And he is still cursing (mostly about other stuff, but about that COM abomination, too).

Can we write the COM part of that code in a better way?

Surely, by making use of C++/WinRT, C++/CX or VC++ Compiler COM Support mode, who still uses stuff like STDMETHOD outside legacy code?!?

And on Embarcadero's side, C++ Builder Component Wrappers.

He is so knowledgable about COM, that in 2018 writes ExitScope manually instead of using either ComPtr or _com_ptr_t.

Can you, for once, provide a simple example instead of dropping lists of enterprise technologies? I'm very sceptic that I want to add a whole new technology to my build just to read a directory path from the system configuration. You might be able to convince me that it's not just a whole lot more boilerplate to maintain, but you won't convince me by parroting these lists over and over again.

I've quickly looked up C++/CX for example, and IT IS A DIFFERENT LANGUAGE WITH DIFFERENT SYNTAX. You have to be kidding me. So am I supposed to learn about this new language and write this one stupid function call that I want to make in a totally separate file and build it with a new build chain?

My position is, the task at hand would be so much simpler if we could just call a stupid function that we link to. I don't know a lot of problems with that approach. Tell me the problems, and tell me why COM solves them without introducing a whole bunch of new problems.

> He is so knowledgable about COM, that in 2018 writes ExitScope manually instead of using either ComPtr or _com_ptr_t.

He is not very knowledgeable about it. He even freely admits that. And he shouldn't be, just to call a friggin' function.

Regarding the ExitScope, look closely what he does there. He's building a defer statement into C++. The defer statement, while I dislike cleverness including classes and macros, is an interesting idea because it is general (unlike ComPtr or whatever, or even unique_ptr). It doesn't work just on COM objects. And it allows you to put the code inline like regular code - just closer to the point where you constructed something (not just objects) so you can more easily see it's properly cleaned up.

The thing is that, while making a new separate class sort-of guarantees proper construction and destruction, it comes at a cost because it artificially compartmentalizes the code. You get this one property (proper cleanup) but you are forced to take that other property. When defining a separate class, it's very hard to see the context in which this thing will be used. And unless it's used in many places, the idea of a clean abstraction usually doesn't come to realization. You'll just end up with a bunch of garbage class code, and when reading it you have to make some guesses about what you can assume. And you end up with two distinct pieces of code that are likely to fall out of sync.

The typical experience I've had with reading such OOP code is that I get dizzy very quickly, and it's extremely exhausting, because at each line I'm reading I have to ask, "where will this line jump implicitly? Which exceptions could it throw?" and if I find the implicit jumps correctly, the whole game just starts over in the next line there. Before I have a rough understanding of what this code does, I'll have opened 123 different files that are each extremely vague on their own.

I've seen people making separate C++ classes like "TemporaryLocale" that just switches the locale to "C" and back because there was ONE stupid problem somewhere. There is no good reason to make yet another class unless there are a couple points where this will be used, such that knowing and internalizing the class pays off for the programmer / reader. (And the proper solution would have been to just not use any of the crap that depends on the locale).

https://docs.microsoft.com/en-us/archive/msdn-magazine/2017/...

Looks like Microsoft freely admits that its previous Technologies like C++/CX are "clunky" to use (which I wager is PR speak for "impractical bullcrap"). Not to mention other technologies which are officially described as "verbose, tedious and complex".

Now it seems like C++/WinRT is the next holy grail and they provide instructions how to port your previous code (which is now obsolete - thanks Microsoft!) from any of the previous technologies to the new blessed one. We'll see how that works out. If I understand correctly, C++/WinRT allows you to write mostly standard-conforming C++, which is an incredible feat! </s>. It might make it easier to write code that is portable between platforms.

Meanwhile I'll just continue using plain C, instead of pulling out the last hair I still have on my head. Because nobody's still given me a reasonable argument what's wrong with plain functions and a linker.

And nobody has been able to give a good argument why we need an obscure technology like COM to achieve a reasonable amount of compile-time and run-time introspection. Why shouldn't we just be able to specify which sets of plain old functions should be used as constructors and destructors, for example? That would allow to write all kinds of OOP languages on top for anyone who is willing to make the associated tradeoffs.

In this context, I don’t think that is a strong argument. The subject at hand is whether it is easy or difficult to write the code that makes that work in a non-C programming language.

IMO, COM more levels the playing field by making it as cumbersome to use from C as from other languages, rather than by making it as easy to use from other language as it is to use it from C.

Yeah, that sounds even more... fun. It had most of User32 an friends (as of 1999 anyways) already wrapped from the Smalltalk vendor, thankfully.

The main problem with C macros is their usage when a regular function would do just fine. That goes for a lot of the cruft in the Windows headers.

Having said that, while I object to most ideas that "C is missing", I do think that better constant expressions and expression macros (functions on the syntactic-expressions level) would be a usability improvement in APIs compared to (token stream) preprocessor macros. Even though in the presence of the preprocessor, constants and expression macros would be somewhat redundant.

One advantage of expression macros is that you do macro expansion after parsing to an AST. You get a valid AST parse with macro expansion if and only if you get a valid parse without expansion. Expression macros are regular AST elements, and are scoped just like regular functions or variables. They don't have a weird textual scoping scheme with conditional compilation or un-/redefinability. Getting rid of that should make it easier to translate them to other languages.

Another advantage is that they allow you to actually know which constructs in an interface are part of the API. An expression macro is always supposed to be used directly by the consumer of the API, or at indirectly through another macro that is. That particular problem of the C preprocessor could in fact be solved if headers cleaned up after themselves, but in practice, preprocessor is just to messy so nobody does that.

Isn't this what rustlang did?

I have to admit that I'm coming around on rust. I still think that their fanboy marketing is obnoxious...but I need to get over my penchant for finding reasons to be angry.

I think they have a variety of different kinds of them. I'm not sure I want to go there. There are already benefits to be had by something that looks basically like function-like macros in C, but where the replacement bodies are required to be valid expressions, and that is scoped like other functions and variables, and only expanded after parsing. I did that for my own (experimental) language and it was very easy to implement and use.

We have two forms of macros: one that’s basically fancy pattern matching, and another that lets you manipulate an arbitrary series of tokens.

>actually know which constructs in an interface are part of the API

Yes, this. I could even tolerate cpp as it is, if I had this metadata. It is relatively easy to generate wrapper.c that simply re-invokes all interfaces, converting them to homogeneous ffi-able form. Like Pascal’s .int files which described everything you want to know as a unit user.

When I worked on my own language that links to libc, it turned out that MSVC has the standard streams (stdin etc) defined as macros. That's really annoying since that means one has to write a C wrapper function to expose these streams. It might have been possible to avoid these macros, but that probably puts some constraints on the allocation done inside libc.

On the other hand, there are some things you simply cannot do without macros. I usually use function-like macros to forward to other functions with additional arguments (like __FILE__, or #arg, sizeof(arg)) which is extremely convenient and allows to cut out a lot of error-prone boilerplate.

From a different language, you can still call the function that is being forwarded to. Now try doing that with C++ templates or whatever...

Hilariously, the C standard requires this.

C11 7.21.1.3: "The macros are [...] stderr, stdin, stdout which are expressions of type ‘‘pointer to FILE’’ that point to the FILE objects associated, respectively, with the standard error, input, and output streams."

You probably meant to point to 7.21.1.3, not 7.21.3. I had to look it up myself, and sure enough, that's true[1]. The C Standard is a magical document indeed.

[1] http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf

Oops, yes, thanks!

Looking at you, PJSIP!

Which is also kind of ironic, because C as universal compatibility only works on OSes implemented in C, as it is the OS ABI, which granted is the case being discussed here.

"as it is the OS ABI"

The C ABI almost never is the OS ABI. On systems with separate address spaces, the OS ABI needs to cross a kernel boundary. In theory, you could use the same ABI for regular function calls, but the overhead wouldn’t make your compiler popular.

>The FFI-library of your language can usually call straight into C-functions but then it turns out half of the C API is implemented in macros which can't be called nor converted automatically

Why is that? Aren't the macros expanded by the preprocessor, and end up as a specific piece of code under a function name in a library?

Or does this mean there's no e.g. X function that uses one or more preprocessor macros inside it, but the very X function is but a macro itself (so it's only "callable" by the public API function name, from C code before expansion)?

> Aren't the macros expanded by the preprocessor, and end up as a specific piece of code under a function name in a library?

Sure, but you don't know the name of that function so you can't write the FFI declaration for it. You could reverse engineer it, but this would be platform and compiler specific, so non-portable and unstable.

Another thing to avoid is functions defined in the headers as `static inline`.

The issue here is that you shouldn't need to interface with C, unless you need to talk to some C Code. You absolutely shouldn't have to interface with C to talk to the kernel.

I come at this from a Linux perspective, but to me the user/kernel interface (sys calls etc.) should be the well defined abstraction boundary. The C library is a useful api making that easy to use, but to dictate that people can only make sys calls through C is silly and ends up with these sort of problems.

There is quite a bit of variation here. Linux and, AIUI, quite a few microkernels, have a defined ABI at the user/kernel boundary. BIOS, a bunch of hypervisors, and some RISC-V M-mode interfaces are similar, albeit with “user” and “kernel” replaced with other execution modes.

OpenBSD and, very notably, Windows instead have a defined library interface. Windows has somewhat stable “system calls”, but only through NTDLL.dll. The real stable interface is kernel32, etc.

There are arguments in favor of both approaches. One significant, if rarely appreciated benefit of the user/kernel approach is that it enables user programs that don’t use the C ABI at all. Go on Linux is like this, and Go and the vDSO have interoperability issues because the vDSO is a bit FFI-ish in Go, and Go’s FFI is not very fast.

A stable syscall ABI is a Linux peculiarity and was never a part of the UNIX tradition.

Here's a talk where Bryan Cantrill describes the tightly coupled Solaris kernel/libc combination as a "welded unit", and mocks the Linux "libc ecosystem" and the complexity of ENOSYS handling in glibc.

https://www.youtube.com/watch?v=TrfD3pC0VSs#t=29m10

Mock it all you like. Old static binaries, old containers, and old chroots continue to work on new kernels. This is a big deal.

There's no problem with directly calling sycalls from OpenBSD from golang or C or whatever else. The issue here is that OpenBSD is adding a new security feature where code addresses that call syscalls must be whitelisted at link time. golang could just create their own syscall interface and linker scripts appropriately whitelisting these locations and this problem wouldn't exist.

This happens due to POSIX. Nothing stops people from using direct system calls but they would rather use a C library for portability reasons.

I'd even say the Linux system call interface is better than the POSIX one. Well documented, no errno. I wrote a liblinux in C and Rust and it's been useful for accessing newer system calls with no libc stubs. The more subtle problem: standard C libraries are incompatible with direct system calls. They maintain a lot of global state which will be invalidated if the program calls the kernel directly. For example, clone() cannot be used with glibc.

The ideal solution is to start from scratch and build a new user space without using C at all.

A function internally called by `errno` macro is part of an ABI, and changing its name will break all already compiled applications for no real reason. Most operating systems provide stable libc ABI (exception: OpenBSD). So it's a rather safe thing to depend on.

> (exceptions: FreeBSD, OpenBSD)

These are some pretty significant exceptions though, especially in the networking-related fields. I don't want the programmes on my router to suddenly not boot after a system upgrade. Especially when I don't have the source code or if the compilers haven't caught up yet.

Back in the day, other languages had their own bindings to the POSIX/UNIX API's. There are standardized bindings for FORTRAN and Ada; not sure if others exist.

The complaint is that the underlying function which returns the thread-local errno pointer isn’t a public API, isn’t standardised, and its name varies from OS to OS (__errno or __error or __errno_location)

This specific case would be resolved if some standard (ISO C, POSIX, etc) defined a public API to do it, e.g. errno_r

(Of course, that doesn’t fix any other cases where some public functions are really macros to private functions, but this appears to be the most significant such case.)

The whole C interface is clunky because C does not support multiple return values. More advanced languages should not be forced to deal with it.

"Should" is as wobbly as errno, no?

> 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

That seems a defensible definition, but also at odds with your usage:

> More advanced languages should not be forced to deal with it

(I'm more in agreement with the definition you offer than your usage.)

Just a tick-- the quoted usage and the definition appear to be from two different people.

Are you sure its not the other way around? Might C not have return values because of the systems it ran on? How would that work today in Linux, for example? Map a page to have some memory shared between the kernel and userspace?

> The whole C interface is clunky because C does not support multiple return values.

C has pointers which can easily enough be used as "out parameteters". In fact the C standard library regularly makes use of these. It's not amazing, but it works fine, and it's usually fairly easy to go back and forth e.g. expressing a stat(2) call is reasonably straightforward.

However there is no way to know that programatically, and no standard way to annotate it either. So in FFI / language bindings context this is really tedious to work with.

well most of the time the return value is just an error integer, while the real return value is a pointer parameter.

My hot take here is that multiple return values are a conceptually complex way of handling tuple return values.

The C API could do this, it just doesn’t.

You can return structs from C functions (note, I don't mean pointers, I mean whole structs). You can pass in structs too. It's not commonly done and is a bit clunky compared to returning a tuple, but it is possible.

What's the difference? My understanding is as follows: A struct is a tuple. It's just some structured memory. Where a tuple would be indexed numerically, a struct is indexed with names of the fields (equiv in Python to NamedTuple)

What happens under the hood would be the same, but in C you have to define a new struct for each return type which is clunky. In ML or Python you can return ad hoc tuples, eg:

    let res, _, _ = fun_which_returns_3_tuple ()

    struct my_result { int res; int other; int third; };
    ...
    struct my_result r = fun_which_returns_struct ();
    printf ("%d\n", r.res);

Tuples can pull their contents from different memory locations whereas structs are laid out continuously in memory (modulo padding if enabled). Of course you could store pointers in the structs you return but that would defeat the whole 'single return' concept, then you may as well return a pointer to a struct which is ugly because it will never be thread safe and is a particularly nice pitfall regarding ownership of that struct.

So either return the whole struct as values or, alternatively - and this is the way most library functions do it - call the function with a pointer to an empty struct which the function then fills. That leaves the return value for error status.

> So either return the whole struct as values or, alternatively - and this is the way most library functions do it - call the function with a pointer to an empty struct which the function then fills.

Mind you, under the hood all your structs will be returned through pointers to caller-allocated memory, unless it can fit into 8 bytes.

“To return a user-defined type by value in RAX, it must have a length of 1, 2, 4, 8, 16, 32, or 64 bits. [...] Otherwise, the caller assumes the responsibility of allocating memory and passing a pointer for the return value as the first argument.”

https://docs.microsoft.com/en-us/cpp/build/x64-calling-conve...

Yes, but that's literally an implementation detail and this could be done in many different ways. The idea here is to gain some amount of expressiveness, if that is the goal then implementation details should be of lesser importance.

Moving structures around 'by value' rather than 'by reference' is wasteful in my view, but if you have to then you should no longer worry about the underlying mechanism until optimization time rolls around.

> Tuples can pull their contents from different memory locations

I'm not sure what you mean by this. Tuples are typically isomorphic to structs with unnamed fields.

> It's not commonly done

It’s really rather common eg several time or socket functions take a pointer as « output parameter », fill it and return an error code.

Not the best but not the worst either.

The Unix API even returned a pair of file descriptors for the pipe syscall. Most ISAs have a way of returning multiple values eg for structs by value. However they fell out of fashion, I think only NetBSD still has a multiple return pipe on some architectures.

The issue here is the interface to the operating system, which should be as universal as is feasible. Having multiple return values would work against that, just as using macros does.

I think it is time to create the new POSIX standard, with better portability and safety/security. Decades of experience can help to design the better API.

New POSIX standards are released quite frequently and participation is quite open, so ... have at it :)

It will never happen, because POSIX is literally the UNIX part of C libraries that did not land as part of the ISO C standard library.

One of the worst offenders is statvfs with its use of __USE_FILE_OFFSET64...

> There is no useful errno variable to load in your own language's runtime after a call to, say, the open() function

FWIW, in rust there is std::io::last_os_error().

libc::perror works too.

The point of the article is that there's glue code needed for this, I think.

It seems like Rust also needs this, either through linker magic or extern C:

https://github.com/rust-lang/rust/search?utf8=%E2%9C%93&q=er...

Reply on news.ycombinator.com