A few facts about POSIX | Hacker News Reader

A few facts about POSIX

vorakl.com

114 pointsvorakl54 comments

> However, there are a number of operating systems, such as MacOS, that are fully compliant with the POSIX standard and can therefore be called Unix operating systems, not just Unix-like.

It's a bit more nuanced than this. Being fully compliant with the POSIX standard isn't quite enough; you have to pass an official conformance test suite and be certified by the Open Group to have passed it, in order to call yourself "Unix", rather than just being Unix-like.

https://en.wikipedia.org/wiki/Single_UNIX_Specification

So there could be (and likely are) implementations out there which are fully compliant with the standard, but have not chosen to get certified, and are therefore technically still "not Unix".

> Being fully compliant with the POSIX standard isn't quite enough; you have to pass an official conformance test suite and be certified by the Open Group to have passed it, in order to call yourself "Unix", rather than just being Unix-like.

Technically, Unix certification and POSIX certification are separate, you can certify as either.

Unix certification: https://www.opengroup.org/openbrand/register/

POSIX certification: https://posix.opengroup.org/register.html

>such as MacOS, that are fully compliant with the POSIX standard

Until you have to gets your hands dirty and find small and bigger faults everywhere. For example, /bin/sh defaults to their ancient (last GPL2 release) bash that doesn't work properly in these two parameter expansion cases:

  $ cat test.sh
  var=
  echo "${var+{\}}"
  cat <<EOF
  ${var+\"z\"}
  EOF
  $ /bin/sh test.sh
  {\}
  \"z\"
  $ /bin/dash test.sh
  {}
  "z"

Second one bit me quite badly when doing heredocs to generate JSON.

Program^WCompliance testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.

-- Edsger Dijkstra (ish)

It really says something about how much Apple cares about restricting its users’ freedom versus implementing functionality for those users that in 17 years they haven’t been bothered to either write their own /bin/sh, fix the ca. 2007 bash they do ship or just ship a newer, GPLv3 bash.

And it really says something about how important it is to license one’s software under GPLv3 or similar!

They basically want you to use zsh... It's been the default shell for a while. If you want a newer version of bash, install it yourself with brew or macports.

What's even more comical is that they also ship dash, as shown in my example! But the reason why is obvious: people using a /bin/sh shebang with bash features.

So I understand, but patching bash to fix the --posix mode should still be done.

I think it's best to consider MacOS not having bash by default.

(The OS shell of choice being zsh.)

That might hold more weight for me if it weren't the easiest thing in the world to install bash! Like literally `brew install bash` will do it, right? That way I'm not reliant on Apple at all, which seems optimal when we're talking about features they don't build and that aren't their focus. I'm very happy to let Apple do the things they're good at and the community do the things they're good at.

Scripts using #!/bin/sh instead of #!/usr/bin/env (ba)sh won't use your shiny new shell. Including scripts deep inside the build system of misc. software that you won't modify.

The only way is to set their strange /etc/private symlink (cf sh(1)) to change what /bin/sh execs, and to do it again after each update which resets it.

I wonder what is the origin of this "/bin/sh" and why it's often assumed that sh is in /bin by default. The standard is pretty clear about that:

"Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH, ensuring that the returned pathname is an absolute pathname and not a shell built-in."

"Furthermore, on systems that support executable scripts (the "#!" construct), it is recommended that applications using executable scripts install them using getconf PATH to determine the shell pathname and update the "#!" script appropriately as it is being installed (for example, with sed)."

Ref: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/s...

So, apparently on systems, where sh resides in /bin, "getconf CONF" will return /bin . However, this is not always the case. For example, on my Fedora Linux it is /usr/bin

  $ getconf PATH
  /usr/bin
  $ ls -l /usr/bin/sh
  lrwxrwxrwx. 1 root root 4 Jan  2  2023 /usr/bin/sh -> bash
  $ ls -l /bin/sh
  lrwxrwxrwx. 1 root root 4 Jan  2  2023 /bin/sh -> bash
  $ ls -ld /bin
  lrwxrwxrwx. 1 root root 7 Aug  9  2022 /bin -> usr/bin

Probably the natural consequence of shebang limitations and the "chicken and egg problem" of env(1) that isn't always in /usr/bin.

If only we could write #!sh to do a PATH lookup...

Thanks for finding the exact POSIX section describing its own failings!

On Gentoo (with the recent merged-usr profile), getconf PATH returns "/bin:/usr/bin", heh.

Apple added /bin/dash in Catalina, and the intention is probably to eventually to use dash as the implementation of sh: https://scriptingosx.com/2020/06/about-bash-zsh-sh-and-dash-....

As for why they haven't done it yet, I'd guess it's just to preserve backward compatibility with old scripts that rely on quirks of bash's sh compatibility mode.

Oh they added it to recovery too. Interesting.

> Until you have to gets your hands dirty and find small and bigger faults everywhere. For example, /bin/sh defaults to their ancient

It’s irrelevant. Neither POSIX nor the UNIX certification (or being a descendant of the UNIXes of old like the BSDs) requires a recent GNU userland (which is, as the name implies, not UNIX).

> > > such as MacOS, that are fully compliant with the POSIX standard

> > Until you have to gets your hands dirty and find small and bigger faults everywhere. For example, /bin/sh [...] doesn't work properly in these two parameter expansion cases:

> It’s irrelevant. Neither POSIX nor the UNIX certification [...] requires a recent GNU userland

Is it irrelevant?

If an implementation passes the conformance test suite, but is buggy according to the text of the spec in a way that is not caught by the test suite, can one really say it's "fully compliant with the POSIX standard"?

Is it the test suite that's the final arbiter of the POSIX standard, or is the text of the spec? Because I thought that the point of having a written spec, ratified by a standards body, is that the text of the spec defines what the spec is. It is the text of the spec that is submitted to the standards body, after all, not the test suite.

I said /bin/sh, not /bin/bash.

They dropped any utility that switched to GPLv3 because of v3's infectious patent clause that would've put their patent portfolio in danger.

The default shell on macOS has been zsh for over 4 years now, and they keep it up-to-date.

We typically distinguish UNIX® which is everybody who has money and wanted to say they're Unix, from just Unix, meaning the things that in practice behave like Unix.

Money talks, so I'm confident that Apple will get to call their mac OS UNIX® until they don't care regardless of whether in any useful sense it's still Unix. On the other hand, regardless of whether some organisation pays money to effect this, Linux is still Unix and so is FreeBSD.

You've probably seen other situations in the real world where the thing you actually care about is distinct from legal status - because our social cues aren't tied to legal status. That doesn't make the legal status worthless, but it is distinct - Obergefell for example is a case where doubtless normal people just accepted that these guys are married - there was a ceremony and everything, but in Ohio at the time the asserted legal status was that they aren't married, therefore the death certificate claims otherwise and hence the case in front of the Supreme Court.

Apparently Oracle has stopped caring, new Solaris releases haven't been certified in a few years, says Wikipedia.

I don't really know what the certification gets you these days other than bragging rights. I imagine in the past, government and corporate customers had check lists that included a box for "certified Posix", but I'm not sure if that is still the case.

It is/was a useful credential to have when responding to government RFPs as governments had purchased UNIX systems and wanted new systems to be interoperable. This might no longer be an advantage in most requests but my guess is there are still some that require it.

There is another meaning. Derived from originally Unix code: so MacOS and FreeBSD are Unix, while Linux is not.

> We typically distinguish UNIX® which is everybody who has money and wanted to say they're Unix, from just Unix, meaning the things that in practice behave like Unix.

From what i've seen Apple gets every Mac Os release UNIX-certified, which is somewhat cool.

But frankly, is it of any actual relevance ? Nowadays it's gnu/linux in (probably) 95-98% of the industry and the rest is likely to be shared across FreeBSD and a few other players.

But in general, I'd say any "POSIX implementation" that's not been certified should not be taken any seriously: as an example, i had a friend that was working with the Zephyr RTOS which claimed to have some kind of posix compliance... Except it wasn't full and it broke in subtle ways. Still a good RTOS (it seems, not my area of expertise) but the "posix compliat" claim should be taken with many grains of salt probably.

Certified Unix. The source link has more details:

    Apple Inc.: macOS version 14.0 Sonoma on Apple silicon-based Mac computers

    Apple Inc.: macOS version 14.0 Sonoma on Intel-based Mac computers

    IBM Corporation: z/OS 3.1 or later with: z/OS 3.1 or later Security Server and z/OS 3.1 or later C/C++ Compiler on IBM Series Processors that support z/OS Version 3.1 or later

    IBM Corporation: AIX version 7, at 7.2 TL5 (or later) on systems using CHRP system architecture with POWER™ processors

    IBM Corporation: AIX version 7, at either 7.1 TL5 (or later) or 7.2 TL2 (or later) on systems using CHRP system architecture with POWER™ processors

    Hewlett Packard Enterprise: HP-UX 11i V3 Release B.11.31 or later on HP Integrity Servers

    IBM Corporation: AIX 6 Operating System V6.1.2 with SP1 or later on Systems using CHRP system architecture with POWER™ processors and 2, 8 or 128 port async cards

    IBM Corporation: AIX 5L for POWER V5.3 dated 7-2006 or later on Systems using CHRP system architecture with POWER™processors

    IBM Corporation: z/OS V2R1 or later with: z/OS V2R1 or later Security Server and z/OS V2R1 or later C/C++ Compiler on IBM zSeries Processors that support z/OS Version 2 Release 1 or later

    The SCO Group, Inc.: UnixWare ® 7.1.3 and later for single and multiprocessor systems based on IA-32 and compatible processors and conforming to PC architecture

    The SCO Group, Inc.: SCO OpenServer Release 5 and OpenServer Release 6 on Single and Multi-processor Industry Standard Intel architecture platforms

Source: https://www.opengroup.org/openbrand/register

> the Zephyr RTOS which claimed to have some kind of posix compliance...

The work is still in progress. It's also important to note, that Zephyr is intended to run on resource-constrained systems. There is not much logic in implementing fork() for example, since technically, there is only one process. So it implements a subset of the interface. Here is the link to the talk from the EOSS 2024 where Chris Friedt shares the current state of POSIX support in Zephyr: https://www.youtube.com/watch?v=nLnmXxXuS6w

Hey, I meant no dissing at Zephyr, i hope it was clear.

What i wanted to get across is that claims about "Posix compliance" might have wide variance in meaningfulness unless backed by a certification.

Compliance with various POSIX standards remains useful and relevant, though the only list of actually certified products I can find[1] includes only RTOSes.

Even without certification, POSIX remains a useful baseline for implementing and consuming cross-platform system services, taken with your suggested grains of salt on the consumer side for any details not specifically documented as POSIX-compliant.

UNIX certification seems mostly only relevant in terms of OS marketing, in the sense that it signifies a vendor's intention to provide a usable POSIX-ish programming environment.

As a target for cross-platform application developers, the list of certified UNIX platforms is at best weirdly specific:

macOS: supported

AIX: supported, though even IBM seems to prefer Linux for new development; IBM-supported Linux runs natively alongside AIX on the same hardware, further limiting appeal

z/OS: supported, but only a "UNIX system" in that it offers a POSIX compatibility layer on top of a system that's fundamentally non-POSIX in design (EBCDIC, expensive process creation, etc.); IBM-supported Linux runs natively alongside z/OS on the same hardware, further limiting appeal

HP-UX: supported until 2025-12-31; Itanium-only

SCO UnixWare and OpenServer: supported (?), but no substantial updates since 2018 (?)

[1] https://posix.opengroup.org/register.html

> From what i've seen Apple gets every Mac Os release UNIX-certified, which is somewhat cool.

I reckon it is because of the lawsuit they got because they used unix in marketing without actually paying for the certificate. see - https://www.quora.com/What-goes-into-making-an-OS-to-be-Unix...

> Linux is still Unix and so is FreeBSD

I thought the *BSD kernels derive (at least in part) from (BSD) Unix kernel code, while the Linux kernel is a different, independently created workalike.

Moreover, typical Linux distro userlands are GNU, and "GNU's Not Unix."[1]

[1] https://www.gnu.org/doc/doc.html

Ironically the macOS kernel is named XNU - X is Not Unix

> The current POSIX standard defines source code-level compatibility for only two programming languages: The C language (C99) and the command language (shell).

AWK is also defined in the standard.

Yes, subtleties are lost here.

sed is also a Turing-complete language.

Interesting! I just wonder... How would you do a conditional branching in sed?

There is a conditional branch instruction in sed that is taken if the last substitute command was successful.

The O'Reilly book on sed & awk is available online for free from various sources (I grabbed a rar archive of it from somewhere).

https://freecomputerbooks.com/The-Unix-CD-Bookshelf-Version-...

It's a good point! I've addressed it.

I referred here to the following definition:

"2.4 Other Language-Related Specifications POSIX.1-2017 is currently specified in terms of the shell command language and ISO C. Bindings to other programming languages are being developed."

But, you are definitely right that some of the tools, defined as standard "utilities", such as AWK, also have their own language. Although, it is only defined in the "utilities" section, https://pubs.opengroup.org/onlinepubs/9699919799/utilities/a... However, "sh" is also defined under "utilities", although it has an extensive language specification that is provided as a separate document.

The motivation for the standard, in the fear of a Sun-AT&T alliance, is also an elephant in the room.

macOS is UNIX certified AFAICT though https://www.opengroup.org/openbrand/register/

It's funny that EulerOS is a GNU/Linux distro that is unix certified, despite GNU standing for "Gnu's not UNIX".

> So there could be (and likely are) implementations out there which are fully compliant with the standard, but have not chosen to get certified, and are therefore technically still "not Unix".

I'm not sure about certification, as compliance was mandated by the US Govt so there may have been one, but Windows NT had its POSIX API layer which I used ages ago to port some Unix network code to it just by changing a couple lines. No tools or anything else, that is, one didn't have a POSIX shell or any usual Unix tool, but the way to say open a socket and initiate network traffic was pretty much the same.

Thanks for pointing this out! I have addressed this important detail.

"The current POSIX standard defines source code-level compatibility for only two programming languages: The C language (C99) and the command language (shell)."

There is also a definition of AWK:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/a...

There is an argument that Awk's language isn't the same category as it is a DSL not intended to be a general purpose language (it is turning-complete, but was not specifically designed as such), and awk itself is considered a complex tool not a language interpreter.

Though having said that, any definition that separates awk from general programming languages like that could probably apply to shell syntax too.

Awk can do more than you think. Read The AWK Progamming Language.

Also:

gopher://hoi.st/1/dl and gopher://hoi.st/0/files/tgl.awk

make up a pretty complete language.

Don't think on AWK as a more powerful grep, you are missing a great mini-language.

There used to be POSIX.5 which standardised Ada bindings and POSIX.9 which standardised Fortran bindings, but both were abandoned (I assume due to lack of interest)

and sed https://pubs.opengroup.org/onlinepubs/9699919799/utilities/s...

and m4 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/m...

and ex https://pubs.opengroup.org/onlinepubs/9699919799/utilities/e...

and bc https://pubs.opengroup.org/onlinepubs/9699919799/utilities/b...

unix isn’t short of programming languages!

Just short of good ones!

I'd always take the old POSIX stuff over whatever gets bought by IBM next year.

Given their longevity, it seems difficult to label them "bad".

Certainly true. I just referred to this definition

"2.4 Other Language-Related Specifications

POSIX.1-2017 is currently specified in terms of the shell command language and ISO C. Bindings to other programming languages are being developed."

I've added a few words and links to other "utilities".

It's missing all the anti facts, why the Unix Haters handbook exists. https://archive.org/details/TheUnixHatersHandbook/

Horrible libc (strings, blocking IO, threads, locale, ...), why all our systems are catastrophically bloated and insecure. Garbage in, garbage out, the tagline of modern unix.

Why you don't have an alt libc that takes safe strings is beyond me.

I do maintain the safec lib actually, but this I just the bounds-checking Annex K, not a proper string library with u8 and identifier support.

I really appreciate that at the top of the article, the author has a link to a bulleted summary of the article.

It's a classy move: it shows respect for the readers' time.

Reply on news.ycombinator.com