I want a better shell (2019)

I believe that the developer experience for all sorts of tooling has improved dramatically over the last 15 years or so, coinciding with GitHub's rise in popularity.

e.g. the plugin ecosystem for Vim or Emacs have improved, let alone options like VSCode. Tools like Vagrant made it easier to work with VMs. Containerised workflows have similarly improved some cases. Languages like Go-lang or Rust seem pretty nice in some cases. etc.

Shells are also the kind of thing that developers use. I don't believe bash is the best we can practically get. -- Though.. I have heard of other, fancier shells; (or cool ideas like https://ambrevar.xyz/lisp-repl-shell/ or http://howardism.org/Technical/Emacs/literate-devops.html). These haven't caught on.

The only thing I've seen after well over a decade of Linux usage on desktop and servers that I've thought "this shouldn't have been a shell script" was xdg-open. But even then when things when wrong I was glad it was a script I could easily tweak and debug, at least.

This, though, I don't understand:

> We can argue about the esotericisms of shell scripting [...] but I feel like it misses the point.

Shells are, among other things, programming languages. How can attentiveness to the suitability of that language possibly miss the point?

I work in FoxPro DOS back in the days.

That is another dimension. Was close to jupyter + IDE + Database Builder + Form Builder + Code Editor + ... and the language to make stuff is NOT a joke.

That is not the only vision that happen in the past but is forgotten (or rebuilded with a massive micro-service/over-engineer/multi-machine version).

/r

So, to me, using the shell is purely because not exist most stubborn people that the one that lives in the unix/c world, that have forced for decades the use of the most arcane and inferior toolchain possible, and always shutdown any attempts of improvements under the excuse "but it works for me".

Is like regulatory capture to 1000x.

/r

You are not the only person. There are HN readers who are not programming language wonks, who actually enjoy using the shell.

Whenever someone today, usually one of said programming language wonks, tries to "improve" on existing shells it never manages to improve over the relative speed/size/simplicity I am getting with ash already. It never feels like its being written by someone who likes the shell; it feels like some programming language wonk who dislikes the shell (thus wants to "improve" it).

Give me a shell thats even simpler, smaller and faster than ash. That would be an improvement. Early era and deliberately "reduced" shells I have tried in place of ash do not have command history. I have tried to learn to work more "deterministically", without need for history but have never succeeded. I think it can be done. (So does the maintainer of the Heirloom Project.)^1 In practice I often edit command lines in vi, using the fc built-in. Why cant I do all editing in vi or ed. I suspect its the addictiveness of the "REPL" workflow.

For whatever reason, I am more motivated to try shells with reduced functionality than I am ones with increased functionality. For any repeated task, taking a long view, non-interactive use is always faster than interactive use, so although a nifty feature like tcsh-style auto-completion might speed up interactive use, a script that automates navigation is going to save time in the long-term.

1. "interactive use. The Bourne shell provides job control if it is invoked as jsh and runs on a terminal. Of course, it lacks fancy features such as a command history, command line completion, etc. But working with these features tends to distract the user's attention. After a familiarization phase, use of the Bourne shell can lead to a more even-tempered, concentrated working style. Give it a try. Seriously."

https://heirloom.sourceforge.net/sh.htm

It's insane that this isn't the default behavior and can't even be configured in 2021!

One that I take issue with is the "everything is a string" mentality of bash. It makes working with structured data harder than necessary, and a ton of stuff I do with my shell involves structured data like JSON. A sane array and hashmap implementation would be nice.

Another thing, which I picked up from elvish, is the concept of separating the textual output of a function from the value it returns. Bash lacks that distinction; the textual output is the value, which makes it annoying to do things like log from a function but also return a value.

I really like Elvish. It's similar enough to bash that I don't feel like the floor has been pulled out from under me, but it makes it dead simple to do something like pull data from a REST API without me having to pull out Python.

The other big pattern I've seen is object-oriented shells like Powershell and Nushell. They seem neat, but too complicated for me to manage in my head while trying to get stuff done. They're a little further towards "programming lanugage" on the range of shell to programming language.

Everyone wants to rush to fix these perceived problems without realizing how some sort of fully automated AI guessing ability about which file to rm -rf is not a good idea, and is not what anybody actually wants.

UNIX systems could offer similar integration, via IPC mechanisms, KParts and DBus like communication, exposing shared libraries directly into the shell and structured data.

Where it fails down is lack of standardization across the landscape, so that everything actually works together.

That said, the language is way better than any traditional Unix shell, and the ecosystem of libraries and scripts is insanely comprehensive. Together, these factors make for an absolutely top-notch scripting experience.

This is definitely one of the the biggest sources of inspiration for most next-gen Unix shells afaict.

I would probably rephrase your statement about object-orientation: I don't think Powershell is particularly object-oriented, but it integrates nicely with object-oriented .NET. What I mean is: when writing extensions to Powershell in Powershell, you may use objects incidentally as containers for structured data, or occasionally to access details in .NET, but you are not creating objects or classes--you're working with functions in a straightforward way, and it's easy to make your functions behave like (take the same options in the same way as) "standard" commands.

I switched (completely) a couple of years ago. The main "use case" that is Powershell's sweet spot for me is handling structured data. A lot of my operations have steps like:

Do some data-wrangling to calculate a URL and request options for some data repository or integrated system API; Perform a network call or look something up in a big file of structured data; Munge and wrangle that data to extract information or calculate summaries; Perform more network calls based on that data and do further munging and calculating.

This is usually discovery-oriented, interactive, ad-hoc lightweight integration, and it's in an environment where I'm doing something with this data (like figuring out a machine list from a couple of CMDBs, cross-referencing that with data from the monitoring system and maybe some lookups for system roles, and then performing commands on them). It's not a great fit for a "real" scripting language because the stuff about processing the data and doing the network calls is intermixed with lots of stuff that the "real" scripting language is not very good at, like interactively discovering what's in the data, invoking a bunch of OS commands and putting things together left-to-right in a pipeline you craft gradually. As your activity matures (you find yourself doing the same thing over and over), you can start factoring your code into reusable units, but not all of it, and Powershell is just super excellent at tasks at this boundary, because it's just fine as a programming language in its own right (much more capable than bash, and with goodies that you reference, like having dependency management) while being at least as good an interactive shell as bash.

As a "Unix guy", I favor a lot of Unix-isms so of course I customize my shell heavily. I can do more powerful things in my prompt much more easily than with bash. I like the more familiar bash-style tab-completion than Powershell's, so I use the PSReadLine module settings to enable that. I use vi keys for command-line editing (with remapped cursor motion keys) and getting this done is easier to accomplish than doing so in bash. I don't like some of Powershell core's default aliases (like curl--I want curl to invoke the curl command, and I'll use iwr instead of curl when I want Invoke-WebRequest), so I disable them.

I do all of my professional work on MacOS and Linux and Powershell is a good shell on both, and almost all of my experience with Powershell is as a Unix shell. I'm not much of a fan of commitment to the so-called "Unix philosophy", but I do think that Powershell reflects this philosophy better than a typical Unix shell and Unix commands. For example, once you learn how to sort the output of ls according to its standard options, all you've learned is something about the ls command. It doesn't help you in sorting a process list from ps. It also doesn't help you in sorting files by something other than size (-S), mtime (-t), atime (-u sort of), "natural version in the name (what?)" (-v), or alphabetically by extension (-X). According to the "Unix philosophy" what is the ls command even doing parsing and sorting version numbers anyway? In Powershell, it's more natural to let Sort-Object sort things, which it can do by any of the item's fields, not just what the ls author provided an option for; and once you know you can do that, you can do it for processes in just the same way, or AWS instances or anything else. Boom. Unix philosophy. (And if you just like using ls, and piping it to sort, that works fine, too, so no custom is imposed on you). It's kind of beautiful.

Our views on what we want out of a shell are always based on the tasks we personally try to accomplish with it, and I don't think the author wants a "better shell" at all, but a better, more scriptable IDE. His "use cases" for the shell are all elementary file management and command invocation and he doesn't really seem clear on which component of his system the shell actually even is. Interestingly, there is a full TUI framework for Powershell and it's pretty cute! I didn't know about it before looking it up after reading this article.

E.g.

    git ls-files --exclude-standard -o | For-Each { Remove-Item $_ }

(Deliberately not using aliases for clarity)

There is a trend of people making increasingly complex and expressive shell languages, but I don't really understand what problem they are trying to solve. Maybe what we really need is a less complex shell so people are less tempted to write complex logic with it.

> But above all else, it's a $SHELL so it works find with all your existing CLI core utils. Which is more than can be said for the likes of Powershell.

Can you expand on what you mean by this? I've been using Powershell as my primary shell for 2-3 years and this statement surprises me. There's no "core util" I can invoke that behaves differently or surprisingly in Powershell so I probably don't understand what you mean.

Interestingly, if all of your machines were to have Powershell core, you can actually do a lot of what you describe over SSH more elegantly with Powershell remoting.

Now my primary interest is essentially whether through an SSH tunnel using minimal tools I can pull the remote resources back to my machine and use a real environment.

"Private - All cloud features are opt-in. Data is encrypted at rest."

Wait... was this even under question? Why would my terminal not be private? :/

Plug - I'd also keep tabs on my project, Terminal Click [0]. It's exploring the relationship between the emulator and the shell.

Nushell [1] and I are talking about possible integrations between the two products.

[0] https://media.handmade-seattle.com/terminal-click

[1] https://www.nushell.sh/

https://github.com/withoutboats/notty/

Interactive shells can be used without opening a graphical terminal. Simple examples are calling :shell from vim, or S in the `ranger` file manager, or entering a docker container, or using ssh.

Terminals can also be used without a shell. For example, I have keybindings for opening a terminal with a file manager, or with vim, or with vim with the clipboard contents and special vim keybindings to handle it, or with nothing on it (a process that sends a stop signal to itself). This last one is to get a new unused terminal window to which I can redirect stuff. For example, with `gdb -tty` I can interact with a program from one terminal, while having another terminal control it with gdb.

They really are 2 separate types of programs that are useful on their own, and their separation allows for more flexible handling, like how separate programs can take control of the terminal at separate times, like how it happens with vim's :shell, etc.

But better integration between shell and terminal could be hugely useful.

Many people have thought of many ideas more advanced than terminals, but the terminals persist due to their superiority at just being simple, flexible, and fast.

It is absolutely possible to “make a better terminal” though. But anyone who does has an uphill battle for adoption.

(context: I was bound and determined to make a better terminal 20 years ago, but eventually gave up on the idea after getting familiar with existing tech and deciding it was good enough for me. Same with Bash, though I still think it’s a horrible language and should be replaced eventually)

hehe, +1

leaning into it harder means understanding it better, the internals, why it works and why it does things the way it does.

Now only if our phones were lisp machines.

"Do everything in Emacs" is an exceedingly short hop from "Maybe just write your own operating system."

It's possible to do "very extensible" and "easy to onboard" -- Obsidian.md is the "actually usable Org-mode" I've been looking for, YMMV.

One of my annoyances with VSCode, which I do think will one day eat the world, is that it still seems the basic level of abstraction is too high. So there's a separate terminal window, which I always have to tell to go away, and which has its own key bindings. What I want is just a text buffer running my shell, that behaves like every other text buffer. Same with the database clients available for VSCode - stop building UIs, just give me text with a process running in the background. The editor is already all the UI I need.

As referenced above, a central database of command, exec status, output, and where executed would be central to this.

THe ability to define sets of machines as clusters and running and returning commands from those in parallel is also essential

The ability to pull, push, edit files remotely...

The ability to store access methods to deliver and return the commands: be it ssh, teleport, (ugh) telnet, kubectl, docker run, aws ssm, etc.

I've written about 20% of that for my current job managing various nosql database clusters. But it is very much ugly and something I'm very far from being happy with, even after about two rewrites

  $ jc ls -l | jq
  [
    {
      "filename": "bin",
      "flags": "drwxr-xr-x",
      "links": 1,
      "owner": "nathanmcrae",
      "group": "nathanmcrae",
      "size": 4096,
      "date": "Jun 28 15:15"
    },
    {
      "filename": "pkg",
      "flags": "drwxr-xr-x",
      "links": 1,
      "owner": "nathanmcrae",
      "group": "nathanmcrae",
      "size": 4096,
      "date": "Jun 28 15:15"
    }
  ]

Everything's not rosy-hunky-dory, each of these have limitations but:

> Can I get a parseable ls?

> Can I get explicit output-as-json flags for all reasonable commands like ps, lsblk, ls, find, grep, etc?

Better than "parseable", you get commands which just return the structured data, for ls (gci), and many other "foundation" commands like ps (gps), etc. I didn't actually think about a structured grep but maybe you wouldn't use it as much with stuff like ConvertFrom-Csv (so you have structured data to select rather than strings to "grep"). Anything you can get JSON or YAML or CSV output from you can just turn into structured data (ConvertFrom-Json) and handle in the same way. Any native Powershell implementation of anything will emit complex data in the same way (e.g. AWS's Powershell tools).

> Can I get a database of commands better than history that shows when they were run and error codes that resulted AND THE OUTPUT, and if it completed for multithreaded/background/etc.

Background job handling is a lot better with Powershell than with bash and allows you to address background jobs by id, get metadata about them and the output, with rcjb (Receive-Job). Your job list is manipulatable with the same Powershell tools as everything else, so you can sort and select easily on things like status or the host on which it executed.

> Can I get a reasonable metashell standard across all terminal apps for doing a lot of what .profile does but isn't tied to a specific file on a specific machine?

It's an interesting idea, hard for me to conceive. Do you mean shell and command configurations that go along with your shell session to other machines? Powershell remoting allows you to establish a session and then mix local and remote commands while retaining your setup and environment, it may partly help. The gotcha: you need to enable the SSH configuration which allows this and (of course) have Powershell installed on the target systems.

It's cross platform, and while the language is Python, it is a modified Python so it's not nearly as verbose. You can write stuff like "cd directory" in your Xonsh script, as well as capture output of commands - No need to import os, shutil, subprocess, etc.

Been using it for a number of years. Really happy with it.

The better idea would not be to try running bash scripts in it, that won't work. The idea would be to use Powershell for your scripts on all three platforms, where Move-Item and Copy-Item do work the same.

In Powershell 'mv' and 'cp' are aliases to Powershell commands, which is why they behave differently. As a Unix Powershell user, I don't want or like that, so I remove them. Scripts can always spell things the canonical way and so be guaranteed of the correct behavior.

There's a huge caveat to this, but it's no less true for Python or any other language, and that is that lots of platform differences are semantic and not addressable by any shell. In my experience you run into these "other factors" pretty quick, but in the domain of build scripts, they might not matter much.

https://archive.fosdem.org/2019/schedule/event/ada_shell/

https://github.com/AdaCore/gsh

You could also use the busybox port of the Almquist shell, if you can refrain from most bashisms.

https://frippery.org/busybox/

* If you use tmux splitting, then mouse selection doesn’t always work right.

* Data with embedded spaces is clunky. Data with embedded line breaks is almost unusable. It can be done, but it’s not the default.

* A lot of work is done with little languages, like AWK, embedded in shell, requiring double-escaping.

He needs a cross platform solution.

Also, magit is horrible in Windows.

0. https://github.com/xonsh/xonsh

Ok wait

Awk and sed are entire programming languages, so like, ok... fair. It hardly seems fair to call grep byzantine to me, since although it is based around a DSL, it's an extremely common one that you'll find in every programming language, including Python. But fine, regexps aren't for everyone.

But pipes?? What's the problem with pipes? Don't you ever compose functions in your code or call map() over your data structures? How are pipes any trickier than the kinds of control flow you might use in Python?