POSIX Is Not a Shell
alganet.github.io
alganet.github.io
If you only use defined behavior and it works, it is compatible.
It’s like saying C99 isn’t a compiler. True, but you can still write C99 code, right?
Sure, but the pojt here is that if we say "Write in X" we generally understand it to mean "Treat X like a standard and don't get too colloquial with the stylings."
Pedantry is worthwhile, but it can be a diminishing returns game.
On the example of 'echo \n' - it's not defined in POSIX, therefore a script written in "POSIX shell" must simply never hit that case.
TFA kinda implies you can't target POSIX shell. That's silly, of course you can. The question is, what tools are there to check for compliance. Whether running on 14 shells is a good such tool - idk. Something specifically searching for POSIX violations might be better.
`local` for example is present in many shells (almost all of them), but they decided to leave it out uniquely because of ksh93 (scope is different). It became undefined behavior.
When the spec was written, ksh was important. Since then, it has only been revised but not updated and I consider it to be obsolete.
So, if you follow POSIX strictly, you then lose local scope on functions, which is more likely to cause bugs and hard to catch with a linter like you suggested. You're left with a broken feature set (on many other angles too) that is not actually practical. Even spellcheck makes concessions.
The blog post stresses this, the difference between POSIX and portability.
If you want portability, testing is better for now. One of the goals of these projects is to more precisely capture a retrospec (what actually works, not what was specified), it's the same thing they did with HTML5.
How so?
Can you give an example that should work according to the POSIX spec but doesn’t in a POSIX compliant shell?
The example in the post is just some behavior the POSIX spec doesn’t specify, so obviously you wouldn’t be allowed to do that when targeting POSIX.
https://stackoverflow.com/questions/11376975/is-there-a-mini...
I use long options in scripts so that you can actually understand which options are used. Short options are for interactive shells since those commands nobody is really gonna look at again.
`echo` in most shells is not `/bin/echo`, it's actually an internal implementation (external echo means forking at every output, which is super slow).
The portability of other cor utils is also a clown fiesta, and most likely the best solution for now is "use gnu", which sometimes feels like "best viewed in internet explorer"-era shinenigans.
* https://unix.stackexchange.com/a/496642/5132
because the problem here is educating people with slipshod ideas about 'sh' being 'the POSIX shell' or (worse) 'the Bourne shell'. Both M. Chazelas and M. Gaigalas are making the point that 'sh' is a language that one aims to write in, not one of the many programs that sort of, sometimes, if invoked in the right way, implement that language; and a subordinate point that people are generally very poor about doing that when yet they insist that they are writing 'POSIX shell script'.
Fun facts: Standardization led by existing practice is not a simple process.
* https://unix.stackexchange.com/a/493743/5132
POSIX/SUS standardization is not a static thing. The long discussed thorny issue of echo has now subtly changed from all of those explanations given about it over the years, because in 2024 the standard was changed.
* https://pubs.opengroup.org/onlinepubs/9799919799/utilities/e...
M. Chazelas's own quite famous 2013 StackExchange answer on the subject has not yet been updated with the change that now incorporates -e and -E into the rule. Amusingly, it was M. Chazelas that raised defect 1222 that caused this change.
osh-0.37$ echo "c:\new"
c:\new
osh-0.37$ shopt --set no_parse_backslash
osh-0.37$ echo "c:\new"
echo "c:\new"
^
[ interactive ]:6: Invalid char escape in double quoted string (OILS-ERR-12)
echo "c:\\new" # with two backslashes
\ " ` $
Nine Reasons to Use OSH - https://oils.pub/osh.html
https://magicant.github.io/yash/doc/_echo.html
Additionally, the ksh family has `print`, which solves some of the issues.
And the story repeats all over again, each shell solving the problem in a different way :) That's one of the main reasons I went for solving the portability problem from the scripting side, not the interpreter side.
$ dash -c 'echo "c:\\new"'
c:
ew
$ busybox ash -c 'echo "c:\\new"'
c:\new
$ dash -c 'echo "c:\\\\new"'
c:\new
$ busybox ash -c 'echo "c:\\\\new"'
c:\\new
It's one of the least POSIX compliant shells. It is derived from the same codebase as busybox ash, but busybox receives more maintenance.
---
In any case, it is pretty sad that sh is in such poor shape than the default /bin/sh on Debian has different behavior in this basic case.
I built OSH to be a set of semantics agreed upon by many shells. I don't think any cross-shell test suites like our spec tests had existed in the past - https://oils.pub/release/0.37.0/quality.html
But there is little coordination among shell authors, and no real motivation to fix the gaps. In contrast, there is A LOT of coordination among JavaScript engine authors, mostly because there are people paid to work on them.
Interestingly, that doesn't allow implementation-defined behavior for "echo -e", for which bash does have special behavior.
case "$({
printf %b "\\061" ||
print -r -- 2 ||
echo -n -e "\\063" ||
command -p printf %b "\\064"; } 2>/dev/null)"
in
1)
alias _printb1='printf %b'
alias _printn1='printf %s'
;;
2)
alias _printb1='print -n --'
alias _printn1='print -nr --'
;;
3)
alias _printb1='echo -n -e'
alias _printn1='echo -n -E'
;;
4)
_printb1 () { command -p printf '%b' "$1"; }
_printn1 () { command -p printf '%s' "$1"; }
;;
'-e 3')
_pdash () { _pd=$1; while :; do case $_pd in -*) echo -n "\\055"; _pd=${_pd#-};; *) break;; esac; done; }
_printb1 () { _pdash "$1"; echo -n "$_pd"; }
_printn1 () {
local _p _r
_pdash "$1"; _r=$_pd
while case $_r in *"$_BS"*) :;; *) false;; esac; do
_p=${_r%%"$_BS"*}
_r=${_r#*"$_BS"}
echo -n "$_p$_BS$_BS"
done
_pdash "$_r"; _r=$_pd
echo -n "$_r"
}
;;
*)
exit 1;;
esac
It exports two callables, _printb1 and _printf1 that escape and don't escape backslashes respectively, are immune to trailing dash gotchas and other pitfalls.
Unfortunatelly, not all shells allow overriding `echo` so I can't proper polyfill when necessary.
I'm going to extremes here, but a simpler version of this can be used for the most popular shells.
But it does feel complex to me.
Personally I just write in the common subset of bash and OSH, which is very large. I never need to support say dash, since every machine that has dash also has bash, which is less broken in terms of 'echo' and so forth.
bash has its own broken-ness, but running under OSH solves that.
And OSH also supports some busybox idioms -- we learned last year that bash cannot run busybox ash scripts on Alpine
e.g. 'chdir' is an alias for 'cd' in busybox, but not in bash. And Alpine scripts use it, so bash can't run Alpine scripts, but OSH can (as of last year).
The pure shell C compiler is an example of that: https://gist.github.com/alganet/2b89c4368f8d23d033961d8a3deb...
It's portable among the most popular shells, but much faster on dash. It's obviously a gimmick, but also a way to stress the interpreters past their usual breaking points.
osh fails at it because of multi-command/block-level alias (I opened an issue with a simplified reproduction).
---
Options are good. Shell is amazing at that, lots of interpreters to serve all kinds of users and use cases.
This post was thought provoking, I wonder, is the hidden argument here that the posix spec for a shell is not well specified if there is so much variance between the implementations?
Or is the fundamental issue simply a matter of history? Both?
My main point is that following the spec doesn't guarantee shell scripts will be portable, which is a common misconception.
That feels like a failure in the spec. Your example illustrates it: echo has unspecified behavior that literally prevents it from being portable.
Is it possible portability is just not a feature of posix?
I started to build something here 12 years ago: https://github.com/Mosai/workshop. Then here 2 years ago: https://github.com/alganet/coral
Now I'm attacking it from another angle, trying to document what I learned methodically and share the tools I made for me with everyone else.
POSIX doesn't guarantee portability. POSIX only guarantees that the things it specifies will work in a POSIX-compliant environment. POSIX makes no guarantees about portability to non-POSIX-compliant environments like Windows' shells or MSVC, the Fish shell, etc. POSIX makes no guarantees about the things it doesn't specify.
This mode is not set when called as #!/bin/bash but is enabled for #!/bin/sh
The full list of legacy behaviors that are altered in POSIX mode are in a URL at the bottom of the bash manual page:
https://tiswww.case.edu/php/chet/bash/POSIX
Ksh doesn't do this.
This would include the MirBSD mksh, and oksh from OpenBSD.
They are much smaller than ksh93 and bash, and I would suggest that their syntax extensions be given preference, as they originally represented ksh88 (from which the POSIX shell standard was derived).
docker run -it --rm alganet/shell-versions:all /opt/mksh_R59c/bin/mksh -c 'echo $KSH_VERSION'
docker run -it --rm alganet/shell-versions:all /opt/oksh_7.9/bin/oksh -c 'echo $KSH_VERSION'
docker run -it --rm alganet/shell-versions:all /opt/loksh_7.9/bin/loksh -c 'echo $KSH_VERSION'
It is easier to understand the POSIX standard with a ksh focus, particularly ksh88.
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
The ksh family is my personal favorite, in particular mksh and ksh93. Both with excelent feature sets. I even made a platformer game that works on them https://github.com/alganet/tuish/blob/main/examples/game.sh (and zsh/bash/busybox too) to show how feature-complete they are.
Shell interpreters are such a broad subject, I could go all day talking about cool things that can be done with them.
I want to get people focusing less on the spec. It's for whoever implements interpreters, not people who write scripts. And there's a gap on that, which I'm trying to cover (with full ksh support, much more than you think).
After David Korn retired, an attempt was made to bring out a new version which failed dramatically, and AT&T reverted to Korn's final release.
SmartOS uses ksh93 as root's shell.
Last commit was 11 hours ago, still maintained.
https://www.reddit.com/r/openbsd/comments/jvbj4s/a_new_appro...
There were core performance problems in the 2020 effort.
It's what Debian and Fedora use, so AT&T doesn't matter anymore. Real-world upstream is the repo I linked, which is maintained. All the 2020 drama is now obsolete.
Blaming POSIX, because a program does not implement it, is childish.
Smells a lot of AI writing.
https://en.wikipedia.org/wiki/Almquist_shell
It's just poetic license.
It's ironic that a post about shell differences glosses over this. Maybe not surprising that slop is sloppy.
I want users to test instead of rote memorizing. And I'm giving them tools:
docker run --rm alganet/shell-versions:all -c 'PATH=; [[ 1 == 1 ]] || echo fail'
But why should I trust your tool if your writing is wrong?
The shell reads its input from a file (see sh), from the -c option or from the system() and popen() functions defined in the System Interfaces volume of POSIX.1-2017. If the first line of a file of shell commands starts with the characters "#!", the results are unspecified.
Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH, ensuring that the returned pathname is an absolute pathname and not a shell built-in. [...]
Furthermore, on systems that support executable scripts (the "#!" construct), it is recommended that applications using executable scripts install them using getconf PATH to determine the shell pathname and update the "#!" script appropriately as it is being installed (for example, with sed). For example:
#
# Installation time script to install correct POSIX shell pathname
#
# Get list of paths to check
#
Sifs=$IFS
Sifs_set=${IFS+y}
IFS=:
set -- $(getconf PATH)
if [ "$Sifs_set" = y ]
then
IFS=$Sifs
else
unset IFS
fi
#
# Check each path for 'sh'
#
for i
do
if [ -x "${i}"/sh ]
then
Pshell=${i}/sh
fi
done
#
# This is the list of scripts to update. They should be of the
# form '${name}.source' and will be transformed to '${name}'.
# Each script should begin:
#
# #!INSTALLSHELLPATH
#
scripts="a b c"
#
# Transform each script
#
for i in ${scripts}
do
sed -e "s|INSTALLSHELLPATH|${Pshell}|" < ${i}.source > ${i}
done
It definitely builds outside docker. It's a musl-cross-make toolchain, you can procure the dependencies locally if you don't like the Docker recipes.
Feel free to open an issue if you feel like that's a challenge. Likely, you can get it to work but checksum reproducibility will be hard without a controlled environment like docker.
> POSIX is a specification. Not a program. The thing that actually runs your script is bash, dash, ash, ksh, yash, or one of a dozen others.
“When someone says ‘write it in ECMAScript,’ they mean well.
“ECMAScript is specification. Not a program. The thing that actually runs your script is Node.js, Bun, Deno, Rhino, or one of a dozen others.”
See how silly that sounds?
... Let's write ECMAScript for portability.
Oh wait, that's exactly babel's headline! It's just so old now that people don't need to say what it does.
Actually funny that the spec drift got solved by polyfills and transpilers.
People say that all the time.
People use JavaScript everywhere.
That simply doesn't happen with portable shell script. There's a lot of misconceptions (people can't distinguish between shell/terminal, shell/coreutils, etc), implementors don't talk to each other.
"People use JavaScript everywhere" means nothing for this discussion, it's about whether the ecosystem grasps fundamental portability questions. The JS ecosystem does (now, after some decades), the shell one does not.
We have babel, core-js and many other libs. It's just a bunch of trick compilations (Ubuntu's dash vs bash page, Rich's shell tricks), nothing actually sturdy.
You can try to bend the discussion to make them closer, but it doesn't survive a critical analysis.
The echo thing would be like if ECMAScript allowed stuff like `"123" == 123` to give either false or true; and then indeed many things would probably break if moved across implementations.
C is the closer comparison, and indeed much software that could easily be portable (and might claim it is) often depends on implementation-specific things like 8-bit bytes, 32-bit int, assuming int8_t/etc in stdint.h exist, twos complement (before C23 at least), arithmetic shift right, etc.