Unix Shell Programming: The Next 50 Years | Hacker News Reader

Unix Shell Programming: The Next 50 Years

micahlerner.com

159 pointsmlerner64 comments

So somehow this paper misses one of the greatest developments in this area, oil https://www.oilshell.org/ - which:

1. has a compatibility mode with sh and bash. it's the only new shell I know with a upgrade path from bash [0]

2. otherwise fixes tons of unix shell misfeatures and in particular fixes quoting hell [2]

PS: one fun quirk of the oil implementation is that it's written in Python.. but actually in a DSL that generates statically typed C++ code, ditching the Python runtime altogether [3] [4] - when you do compile Oil, you just get a C++ tarball (somewhat like fftw, a FFT package written in OCaml but which generates C code; fftw developers write OCaml, but fftw users compile C)

[0] https://www.oilshell.org/why.html

[1] https://www.oilshell.org/blog/2020/01/simplest-explanation.h...

[2] https://www.oilshell.org/release/latest/doc/idioms.html

[3] https://www.oilshell.org/blog/2021/01/why-a-new-shell.html#w...

[4] But actually I think there's still work to do to remove the CPython dependency at runtime, https://github.com/oilshell/oil/issues/636

That paper misses a lot. Including:

- murex https://github.com/lmorg/murex

- Elvish https://github.com/elves/elvish

- NGS https://github.com/ngs-lang/ngs

- Powershell

- And any language REPLs turned into shells, such as Python, LISP, Lua and others.

The author did say there is more to follow though so maybe the scope of this article was a little more closed by intention.

Great choice of name when there’s already an oil company called Shell.

Presumably that won't matter so much in 50 years, since either Shell will be bust, or civilization will have collapsed.

This is a common reaction, looks like I have to make this web page :) https://news.ycombinator.com/item?id=29293539

FAQ about the name: http://www.oilshell.org/blog/2019/06/17.html#why-is-the-proj...

I never thought I'd see a fftw reference again. I used it almost 6 years ago for a astronomy telescope data processing library, for convolution. I had no idea!

(author here) Thanks for nicely summarizing the project!

I didn't know about fftw, but another nice analogy is TeX by Donald Knuth! TeX is actually written in WEB / a subset of Pascal (H-Pascal I think), and then translated to C. That is, it's NOT typically compiled with a Pascal compiler (I wonder if anyone even maintains those?). This came up a few times on Hacker News and I got some interesting details.

I wrote yesterday that I want to hire a Compiler Engineer to help with the C++ translation:

https://news.ycombinator.com/item?id=29605304

It's already half done, passing ~1100 of 1950 spec tests, so I believe it's very feasible. But it needs someone who can concentrate on it full time.

I might as well post the draft of the "job listing" to get people thinking about this. I think the challenge is finding the right person rather than funding it (I'm applying for funding now)

https://github.com/oilshell/oil/wiki/Compiler-Engineer-Job (rough draft, you may see polished versions of this later)

One sentence description: write an ~10K line translator for statically typed Python, and a ~5K line C++ runtime, and get a free shell out of it! (over 5 years of painstaking work on the semantics!)

The input is ~36K lines of Python the output is ~100K lines of C++. This is a very doable project for an experienced compiler engineer. :) So if anyone knows a good candidate, send them my way at andy@oilshell.org.

----

Also, I (virtually) attended the panel where this paper originated:

http://www.oilshell.org/blog/2021/06/hotos-shell-panel.html

It was basically a call for researchers, hence lesser focus on existing and new implementations.

I think it's good to publish future-facing ideas (almost all of which I've read), whereas Oil is a lot of engineering work!

It seems odd that any paper on this topic (I read both TFA and the paper it references) would not mention PowerShell once, nor the broader issue of how the Unix shell uses text streams as the only intermediary format between pipeline components.

PowerShell might not satisfy: "Universal composition, Stream processing, Unix-native" points. Have you tried to pipe some bytes via PowerShell? Binary pipeline looks complicated https://stackoverflow.com/questions/33936074/decode-powershe...

I'd suspect the OP would not say PowerShell is perfect or should be the *nix future, but that it's worth evaluating the ideas it brings to the table

50 years is quite the time horizon after all

As someone who moved to Linux full-time, professionally and personally, there are times I do miss PowerShell. Never used anything like it. To tab complete "apis" and complex data structures was very enjoyable

For better or worse, it's what I and my team used to manage a decent size VMware cluster of a hundred hosts and 6k VMs ( I think, it was a very long time ago)

Wouldn't go back, but there are aspects I miss

powershell (core) is on linux btw

Powershell is available on Linux. Albeit it doesn't always play too nicely with existing CLI tools.

There are also other typed shells that appeal to yourself which aren't Powershell but do have behaviours similar to Powershell and which do play a lot more nicely with the wider Linux / UNIX ecosystem:

- murex https://github.com/lmorg/murex

- Elvish https://github.com/elves/elvish

- NGS https://github.com/ngs-lang/ngs

I'm the maintainer of murex so happy to answer any questions on that shell here. I do know the the maintainers of Elvish and NGS pop on HN too.

It seems odd that everytime there is a discussions about shells on UNIX, an ad to powershell comes in. Powershell might have its virtues but: 1)One cannot rely on it for system administration when limited resources are available. 2) even on windows is slow to start. 3) documentation is ... where ?

>everytime there is a discussions about shells on UNIX, an ad to powershell comes in.

The gp (Paul) is not "advertising Powershell" and he's not recommending people switch to it.

Instead, he's saying that it's strange that an academic paper talking about future concepts doesn't even have a cursory survey of what other popular shells have done.

Here's an example of that style of writing where other platform technologies are mentioned: Joshua Bloch is a ex-Sun employee and Java advocate proposing new syntax for Automatic Resource Management and in his note, he mentions what C# and C++ did: http://mail.openjdk.java.net/pipermail/coin-dev/2009-Februar...

It shouldn't have to be said that citing C# & C++ does not mean it's an "ad for C#".

When a new 2021 paper about future of shells has no mention of zsh, Powershell, etc by the authors, it's reasonable for readers to wonder if the team has overlooked what others have done. E.g. Powershell was designed by ex-UNIX admins.

The paper is 7 months old and has been submitted a dozen times. Most discussion is here:

https://news.ycombinator.com/item?id=27378444

The title is promising, but when I read the paper half a year ago I found it disappointing, no practical value. Academic considerations to get a conference paper.

The shell is first and foremost a user interface. Its entire purpose is just to help users use the computer. A huge amount of its functionality is in dealing with making it convenient for people to do work - from looking at command history, to skipping back and forth along a text line, to dynamically changing a prompt, to auto-completion. Think of it as an IDE.

Shell 'scripting' is only supposed to be a way to automate that IDE. It's not supposed to be a programming language. Literally the purpose is that you're sitting there at your terminal and you think "boy, I would really love to rename 100 files in a certain way". At first you write a quick one-line series of commands with a loop. But the intention is to eventually write a real dedicated program (like the `rename` command) so you don't have to write a crappy line of "scripting".

Shell scripting should remain limited in functionality, and be tailored to helping someone use the IDE. If you want a systems programming language, then develop one. Go crazy and develop new ideas and push the boundaries of computer science. But please don't ruin the shell experience by trying to solve some unrelated problems with it and end up ruining the simple end user experience.

I am a bash expert.

This paper reads like it was written by clueless academics with no real world experience.

Multicore bash already exists and is seldom used, through the "parallel" cmd.

Other points address no particular issue, and would actually make the language more painful to use.

Bash has its limits, which pushes you to use a full fledged language if you attempt to walk past them. It's a good system.

Could you tell us, why would you choose to use Bash compared to other programming languages if you are making complex programs?

I think your parent poster is implying that they don’t use Bash for complex programs.

I believe that was the implication, too.

One's level of expertise will determine what is "too complex for $SHELL". When that point is reached, it's time to switch to something else, not time to add more features to $SHELL.

(If, when something becomes too complex for $SHELL, you switch to assembly, you might be too much of a $SHELL expert!)

Not OP, but I would say that we, regular Bash users, don't really write complex programs in Bash, even though it depends on what you call "complex". Some of our programs can be pretty long, and we can do more within Bash than some programmers know (eg. the arrays/associative arrays features, the vast parameter expansion options etc.), so no, we aren't bound to some 100-line-max rule, if that's what you had in mind.

Bash (and other shells) are very good at working with other programs and using the filesystem. It's true you have to know many idioms, but once you're there, you can be quite productive for these kinds of tasks.

For me, anything that I couldn't write without a guide is too complex. Like a function. Or a conditional.

Subjectivity of what is complex is the problem here, which means that regular Bash users regularly write programs that are complex to me. And I'd much rather use something with less footguns, even if the good parts of the shell become less easy.

So true. The shell encourages one to reason about and simplify workflows by leveraging programs that already solve the complex problems. I'm continually amazed at how many lines of code I can replace with a Bash one liner.

What usually happens in my case is I start off writing a simple program in Bash, then it unexpectedly becomes complicated as I add more and more to it.

Like I said in the last paragraph: I would not.

If the bash program evolves to span multiple files and lots of lines, I usually switch to ansible/ruby/clojure

I'm also a bash expert and I agree!

I wrote 75 lines of bash to do failure-resistant streaming parallel computation on heterogeneous distributed systems. "parallel" was very close to doing what I want, but it wasn't particularly hard to do it myself without it. I fire it up, peg every core on as many spot instances as AWS will give me, then shut them all down.

I regularly pass data structures through pipes, and "jq" does everything I have needed so far.

The tools can always get a bit better, but I don't need my duct tape to be any smarter, KTHX.

what makes one a bash expert? genuinely curious if i could call myself one.

What makes me an expert is the amount of computer books I have near my desk. Sort of like how the enchantment table works in Minecraft. I am also a bash expert because I own an O’Reilly book on the subject I bought at a Borders 20 years ago at retail price. :)

I am a bash expert as well then, and probably the only person insane enough to have a web framework, complete with http server written in bash.

I'd say you were both masochistic and way beyond just being an expert in that case.

Bash Technical Fellow?

10 years ago my colleagues were astounded by my airtight bash scripts which formed the backbone of our deployment pipeline. I've come a long way since then, and I still learn new things about it regularly, so I guess I'm a bash expert.

I’m also a bash expert but I disagree with your point.

Bash was good enough when being a sysadmin meant your structured data was largely just flat lists of plain text. However the industry moved away from that model years ago. And these days it’s more pronounced than ever with myself spending more time in YAML, JSON and tables thanks to modern DevOps tooling than I have done in the 2 decades of experience I had prior.

Sure, jq is amazing. But if I’m having to reach for jq in nearly every pipeline then one has to question why the shell can’t just understand JSON natively itself. And why should I have to learn a different tool with their own formatting language for working with YAML, CSV, XML, S-expressions and so on. Why can’t I use the same shell primitives regardless of the underlying data format is?

This is why I switched away from bash. I’m enough of an expert in bash to use it for DevOps everyday but I’m also pragmatic enough to realise that other shells actually do offer quality of life improvements. This doesn’t mean those shells have to be fully fledged programming languages - I’m really not suggesting that we should be using shell scripting for anything complicated. The problem is the stuff that isn’t complicated has evolved to become complicated in bash because the nature of our work has shifted away from what bash did well.

But each to their own. I wouldn’t want to impose my workflow on others anymore than I’d want others to lecture me about theirs. I just wanted to offer a counterpoint from someone who is also a seasoned sysadmin (and more recently “DevOps”).

I see your point, and you're right - it is a matter of personal taste. That's just where I draw the line as to what the shell should understand. I'm mostly in K8s these days where JSON and YAML are interchangeable so I just need jq (or yq, which is just a jq wrapper).

I'm curious, though, what other shells offer 'quality of life improvements' ... and can I convince my colleagues to install it everywhere?

> I'm curious, though, what other shells offer 'quality of life improvements'

To be honest most alt of them do. Bash has enough footguns to incapacitate a small army. Seasoned users like ourselves know how to mitigate them but using a shell that doesn’t require a conscious effort to support spaces in file names and other really basic things is one less piece of mental overhead I need to consider when working.

> and can I convince my colleagues to install it everywhere?

I’m not sure what you mean here. Are you trying to convince your colleagues to switch so you all use the same shell or saying you still spend a lot of time in remote SSH sessions?

Either way, it sounds like you’re probably best sticking with Bash if those are your requirements.

My remote workflow is largely API driven (serverless) and the few times I do need to SSH into a remote instance it’s usually just to perform basic tasks. The real meat of my console work is done locally these days.

When you say "switched away from bash".. what did you switch to?

Murex https://murex.rocks

The fact that you can cobble together some parallel execution is missing the point.

Bash is an error-prone insane mess. It's not "a good system". Frankly it's so obviously terrible that I wouldn't trust the opinion of anyone that calls themselves a "bash expert"! It's like being a punch card expert.

Actually that's unfair to punch cards. They were well designed at the time. Bash was born deformed.

As a daily shell user, I want a shell that uses even less resources, and perhaps even does less. Otherwise I am going to use the default scripting shell. Today, that's an Almquist derivative as it has been for 20+ (at least, for NetBSD, Linux adoption was more recent). Never do I see proposals for a "next generation shell" which uses less resources than Almquist and can do all the same things. Anyways, Almquist shell isn't going away anytime soon, not for at least another 50 years. IMHO.

In order to use less resources, features will have to be removed.

The pdksh version of the Korn shell was available to the POSIX shell standardization committee. It was not chosen because it was too large, with too many features.

Another round of less-capable tools does not seem productive.

I dunno, I've never had any complaints about bash's performance ... but I do appreciate the idea of a simple performant shell, as long as it doesn't break anything. I was annoyed to find many #!/bin/sh scripts broken in Debian 11 until I update-alternatives'd it to bash. There goes my performance, but hey presto now things work!

> As a daily shell user, I want a shell that uses even less resources

Why is this important consideration for you? What's the use case?

Resource conservation. Computers I use are often older and/or have limited resources. I want to use available resources for other things.

Example: http://cdn.netbsd.org/pub/NetBSD/NetBSD-current/src/distrib/...

Unix shell isn't usable by normal public. It's just a stash of random symbols everywhere with unintuitive semantics. You'd loose your brain if you begin writing one. There are so many quirky considerations you've to take in mind to handle for all cases just gets even more complex then remembering C++ rules.

The substitution and shell-quoting are the things that piss me off. Why in 2021 would ever want all that legacy cruft? I better write a Python script that is far more readable and intuitive than figuring out all the symbol combinations in shell programming. We deserve better, don't we?

Shell programming is like necromancing, one simple mistake leads to big disasters. Also, these shells never point out the actual mistake, like are very unhelpful.

I hope not. I was hoping that Unix and its shell would die out at some point. But it is a hydra that will live on and haunt our dreams.

FWIW, I have switched to Powershell & Python for most of my work. I only use Python when I have to share scripts with colleages, though. Powershell still beats Python for scripting.

I would like a SAFE distributed shell - allow me to run commands seamlessly on remote machines as if they are on my own.

Check out op https://docs.shoreline.io/op op> hosts | filter(cpu_usage > 80) | `pkill high_cpu_proc`

Powershell. At least you have actual objects returned and don't have to parse stdio

I'm surprised to see no mention of nushell [0] I've tried it a while ago and seemed really promising

[0] https://www.nushell.sh/

Thanks to the author for posting this. Thoughts, in no particular order:

Repeatability:

- We already have a dataflow graph, it's called makefiles

- Going from shell commands to Ansible playbooks can be made easier

- Declarative, idempotent commands

- Integration with version control

User friendliness:

- GNU standards. If I cannot use --help and --version, I get angry

- Backward compatibility. Ripgrep, I'm looking at you

- Autocompletion is still in its infancy. Co-operative autocompletion is easy, and can be done by utility authors. What are the options (no pun intended) when the author would not update an autocompletion script? fzf and its kin greatly enhance user experience

- Performance. Accidentally quadratic stuff is a Cthulhu-scaled abomination

- Fancier prompts without needing to install extra software or add a bunch of PS1/PS2 scripts

'Novelties' (not so much, other OSes have had them before):

- Coprocs

- Easier parallelization

- Improvement in job control - freezing and thawing jobs, moving jobs to other hosts etc. K8s/kubeflow and friends

Security:

- Better integration with secret managers

- History cleanup when passwords are passed on the command line

> Backward compatibility. Ripgrep, I'm looking at you

Is that really that important? I do have a few POSIX incantations ingrained into my muscle memory, but it’s not a good thing imo. Is that 4 God knows what flag and strange order really that important? Maybe add a separate backward compatible mode when issued through the old tool’s name, but I would like at least a partial revamp of the frankly, not too ergonomic POSIX tools. While Powershell can sometimes be more verbose, I would prefer a similar convention.

Definitely important.

It is a damper on gung-ho, brazen disregard for existing code/user base.

It prevents code base rot and saves money for corporations/governments.

It forces developers to adhere to the principle of least surprise, and ensures smooth user experience. Mind you, Powershell beats POSIX on readability hands down, and autocompletion would make longer, more readable options much easier to write. Still, the easiest way to increase adoption is to meticulously copy conventions from predecessors.

Compatibility mode would be enough for most purposes if it's maintained and not dropped three years later in the name of progress and limited maintainer attention span.

> Backward compatibility. Ripgrep, I'm looking at you

What specific backward incompatibilities have caused you problems?

Reply on news.ycombinator.com