The Unix API is more than just system calls or Posix (2018) | Hacker News Reader

The Unix API is more than just system calls or Posix (2018)

utcc.utoronto.ca

77 pointsgoranmoomin22 comments

> As people have found out in the past, the real Unix API is the whole collection of behaviors and environments that Unix programs assume. It isn't just POSIX library calls; it's also the shell and standard utilities and files that are in known locations and standard capabilities and various other things.

But... "the shell and standard utilities and files that are in known locations and standard capabilities and various other things" are also specified by POSIX.

(Historically, they were the "POSIX.2" standard, and the libc/syscall library functions were in the "POSIX.1" standard, but since 1998 when POSIX became the ratification of Single Unix Specification version 2-and-later, they're all just in "POSIX".)

So... yes?

> Unless you intend for your program to be narrowly and specifically portable to POSIX or an even more minimal standard, it is not a bug for it to rely on portions of the broader, de facto Unix API. It's not even necessarily a bug to rely on APIs that are only there on some Unixes (for example Linux and FreeBSD),

Why would any of those things be considered a "bug"?

> for example, can you implement a compatible and fully capable Bourne shell using only public Unix kernel APIs, or at most public C library APIs?

What other APIs does the author think that Bourne shell writers used?

I feel like I'm missing some broader context for this post.

>But... "the shell and standard utilities and files that are in known locations and standard capabilities and various other things" are also specified by POSIX.

Unfortunately, it doesn't specify those known locations. It explicitly declines specifying that /bin/sh exists.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/s... says:

> Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH, ensuring that the returned pathname is an absolute pathname and not a shell built-in.

and it doesn't specify the location of getconf either, so you have a chicken-and-egg problem where you need the standard $PATH to find getconf to get the standard $PATH.

It seems to be a convention those days to use `#!/usr/bin/env sh`. At least it's a single entry point. Why can't we just write `#!sh` I never understood.

Another interesting snippet from your link:

"Furthermore, on systems that support executable scripts (the "#!" construct), it is recommended that applications using executable scripts install them using getconf PATH to determine the shell pathname and update the "#!" script appropriately as it is being installed (for example, with sed)."

So there could be systems which does not support shebang yet claiming to be POSIX?

Why can't we just write `#!sh` I never understood.

It's because #! is handled by the kernel, but the process environment is not parsed by the kernel.

Parsing PATH should not be that hard.

Sure add arbitrary user input string parsing to the privileged kernel in a place where programs are launched. The CVE database is pretty sparse as it is and could use a good churning.

So, assuming the string is parsed safely and securely, does the kernel have enough semantic knowledge of paths to know which service to pass it to? Are the services guaranteed to be loaded and running at all times? Or is there an underlying architectural assumption that the kernel is a Linux-style monolith and POSIX no longer applies to micro- and nano-kernel systems?

It isn't... now. On a PDP-11 with 40 kW of RAM, on the other hand...

If you have a shell:

# command -p getconf PATH

If you're writing C code of course you just use the library function.

This is particularly an issue with Solaris where the POSIX binaries are in their own directories.

> It explicitly declines specifying that /bin/sh exists.

Huh. TIL. I've worked with POSIX for a couple of decades now, and I was sure that it required `/bin/sh` to be a valid path to the shell.

Thanks!

It's a bit ambiguous, people often mean the 90s POSIX versions when they say POSIX, even though later some rebranding/republishing of SUS happened under the POSIX name. SUS is the less ambiguous term to use I think when talking about the newer stuff.

"Why would any of those things be considered a "bug"?"

As stated in the sentence, it would be considered a bug if the program was intended to portable.

Why would it be a "feature" to write non-portable programs that only compile on some but not all UNIX-like operating systems.

I can think of some reasons but none of them benefit end users who like the freedom to use a variety of UNIX-like operating systems.

Why would it be a "feature" to write non-portable programs that only compile on some but not all UNIX-like operating systems.

Because you want features that require platform-specific APIs and two Unix-like systems (Linux and Darwin) have 99.9% of the marketshare among your audience?

With this attitude Docker wouldn't have happened, because we'd still be bickering about a standardized API for container primitives, because someone wants to run containers on the Lites kernel on a PA-RISC machine.

That still doesn't make it a feature. If it were as easy to make it portable as not portable (which it often is not), you would choose portable.

Context is conversations where people assume that POSIX is the spec for Unix, and some other OS is POSIX, so just re-compile already... or hey, we won't have to rewrite a lot of tooling for some other OS because we used POSIX OS... or it will just magically run because POSIX.

Incidentally, a ton of GNU utilities aren't POSIX unless you set the POSIXLY_CORRECT environment variable...

> (Historically, they were the "POSIX.2" standard, and the libc/syscall library functions were in the "POSIX.1" standard, but since 1998 when POSIX became the ratification of Single Unix Specification version 2-and-later, they're all just in "POSIX".)

Ohhh, that's how Windows could be POSIX-compliant back in the 90s while not appearing like it at all. Thanks.

> It isn't just POSIX library calls; it's also the shell and standard utilities and files that are in known locations and standard capabilities and various other things.

POSIX has standardized the shell and utilities since before 1990. Of course, not all of them. As far as locations go, you have to rely on PATH; systems do not agree on what exactly is in /bin, /usr/bin and /sbin.

> A 'Unix' without a useful $HOME environment variable and /tmp may be specification compliant (I haven't checked POSIX) but it's not useful, in that many programs that people want generally won't run on it.

Hits for the HOME variable in POSIX:

https://pubs.opengroup.org/cgi/susv4search.cgi?KEYWORDS=HOME...

Under the Shell Command Language it is documented that HOME is "[t]he pathname of the user's home directory. The contents of HOME are used in tilde expansion (see Tilde Expansion)."

/tmp is a de facto standard. POSIX encourages applications to check for the existence of a variable called TMPDIR, but doesn't require implementations to provide that.

There are functions which abstract creating a temporary file or directory. Some of them need a template, some don't. It is possible to write a program which uses /tmp for temporary files, yet doesn't refer to "/tmp".

Interesting that this person would write a post about how much is supposedly not in POSIX, but at the same time saying "I haven't checked POSIX".

Rings of "did you know that 'gullible' is not in the dictionary? It's true, look for yourself!".

I can see his point but it seems to rely on redefining what the term "API" means. You could as well say "The Windows API isn't just .NET - it's Win32 and a whole lot of other things". Which is kind of true, but we don't usually call all of those things an API. We call some of those individual parts APIs, just like we do with UNIX. And other parts we'd call filesystem conventions and so on.

I think he's arguing that you need to know all of those things to program with the OS and he's right, but that's still not how we conventionally use the word "API". It's more of a "standard" and the POSIX standard actually does define a lot of this.

The API is the tip of the topological order of interfaces provided by the platform. For example, .NET would NOT be Windows' API because .NET depends on WIN32 (or even more appropriately, NTDLL) which makes .NET important but non-essential. It doesn't make sense to define "the API" as including all the layers of turtles that wrap the canonical one. Just like we wouldn't say that the UNIX userspace utilities like cp/mv/ls are "THE unix API" because those tools all depend on the system calls.

On UNIX the system call interface is THE api for the UNIX operating system, and it IS what people commonly refer to as the C API. It's just a more formal definition of the C API and it is defined by the Open Group. It's abstract and language agnostic because it's provided by well-understood and widely encoded binary interfaces which have been defined to nearly exactly model the abstract so-called C definition. Therefore when we say "C API" we aren't necessarily including things like all the manipulations possible in an implementation intended only for a single language like C, e.g. glibc header files or a thunk that irons out some weird piece of legacy assembly technical debt like 386BSD's affinity for the carry flag. Because unless kernels are willing to implement the same degree of SYSCALL instruction fascism that Microsoft implements with NT then no UNIX vendor really has the right to make that claim that the ordinals they copied from the System V codebase. So people who think that "system Libc" and "system API" are the same thing are generally missing the forest for the trees. Dynamic shared objects are stupid and they aren't systems. A libc is just a tool for interfacing with real system apis.

This is kind of an odd take because POSIX is purposed to make application development across Linux and Unix easier by providing reliable behavior. Saying, "my OS is better than the high level standard because it does more" makes me go, "Okay".

Also, I'm not sure the author has ever actually read POSIX?

If we could finally semantically split the signals as those which can be asynchronoulsy delivered and those which sort of cannot (segfault)... that without extra and bonus kludge.

Reply on news.ycombinator.com