Hacker News Reader

Not to mention how dangerous it is to use. Accidentally type the > symbol? Enjoy overwriting whichever file came after in the command. Output to an existing file? Old file is gone. Accidentally put a space the wrong place? Gone.

And somehow people think mastering this careless design is something to be proud of.

Can anyone name any other popular piece of software that is that relaxed about irreversibly destroying your data?

Pretty much every RDBMS on the planet (and some no-SQL databases too). :)

I jest but I do agree that Bash et al have their warts. Seriously obscene warts that only make sense when you look at the language from the content of 40 year old systems. But these days there are other options if you want the power of a command line but without (most of) the warts of traditional shells.

A circular saw can easily ruin a piece of work if your not careful, and it could even chop off your hand. We don't tell carpenters that they shouldn't use saws because they are dangerous. There are lots of jobs that are work with dangerous tools.

If a manufacturer could easily design the circular saw to be safe (á la SawStop) with little added cost, but decided not to, I think most people would consider them grossly negligent.

Huh.

I didn't know about SawStop before, so your comment sent me on a wiki-hunt.

"Gross negligence" seems like a huge understatement for all the rationalizing and lobbying manufacturers did to avoid having to use safety technology.

https://en.wikipedia.org/wiki/SawStop#Opposition_from_trade_...

A saw is <<hard>>ware, a physical thinghard to change and improve, while a shell is <<soft>>ware, supposed to be easy to change and improve.

"Not to mention how dangerous it is to use. Accidentally type the > symbol? Enjoy overwriting whichever file came after in the command."

At least in zsh, you can prevent such mistakes by

  setopt NO_CLOBBER

It's saved my bacon many times.

Also, from bitter experience I've forced myself to get in to the habit of never typing:

  rm *

but instead always doing:

  cd ..
  rm foo/*

It forces me to be more conscious of what I'm deleting.

I have `alias rm="echo DISABLED"` in my shell rc, and I have `trash` defined as a big shell function that safely moves stuff to the x desktop trash bin without clobbering anything. It's mildly irritating, but it gives me incredible peace of mind.

> in zsh […] setopt NO_CLOBBER

In Bash it’s

  set -o noclobber

or

  set -C

Those work in any POSIX shell, and also zsh.

It seems you are correct:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...

If you are that concerned about this specific problem, shell allows you to "set noclobber" (or "set -C") to not overwrite existing files.

Yup, I only learned about that while writing https://www.oilshell.org ! I'm sure that has been around for decades; shell also has discovery and documentation problems :)

> shell also has discovery and documentation problems

Only because people refuse to read the specification. The POSIX Shell & Utilities volume is exceedingly clear and straight-forward. (I vehemently disagree w/ its characterization in that paper.) It helps if you use the frames-based HTML version to make it easier to navigate and explore the various utilities, and especially to see how the syntax specification is broken down--while that section is short, it's difficult to appreciate without explicitly seeing how the subsections are related.

I only code in POSIX shell because I really can't be bothered to remember all the minutiae of other shells, almost all of which provide strong POSIX compatibility, Bash especially. I've never lamented the lack of arrays; but I regularly wish that POSIX would adopt ksh-style coprocesses so dash would have them. The shell is really all about stitching together processes (including subshell contexts), and coprocesses fill a conspicuous gap in functionality.

I've never used ksh-style coprocesses so I'd be interested to hear about some use cases for them, and what you like about them. Examples would be even better -- I've never seen them, and I've looked at a lot of shell.

Bash has a different implementation of coprocesses that doesn't seem to be used much, and is very limited. I believe there can only be one coprocess active at a time, which is very un-shell-like! I got an e-mail once from someone who was using them, and I felt it could be achieved by other means.

I have a good idea coprocesses in Oil which basically is a startup time optimization. The coprocess will basically take argv over a unix domain socket, and have other commands to change the process state. It solves "the problem of VMs and containers that start slowly".

https://www.oilshell.org/blog/2021/04/release-0.8.9.html#wha...

Unix domain sockets are already used for the headless shell in Oil (more coming on the blog about that)

https://www.oilshell.org/release/latest/doc/headless.html#im...

The simplest description I've found is at https://www.dartmouth.edu/~rc/classes/ksh/coprocesses.html, which also provides a couple of examples.

Basically, you start a coprocess using the "|&" operator:

  somecommand |&

This starts somecommand asynchronously, with stdin and stdout attached to anonymous pipes with descriptor numbers that can be referenced by a literal descriptor name, "p". So running

  cat <&p

will shuttle the output of the somecommand coprocess to stdout.

"p" always references the most recently created coprocess, so to juggle multiple coprocesses you have to dup "p" to explicit descriptors:

  exec 4>&p
  somecommand2 5<&p |&

That's not as great as having named references, but it's rare to need to independently juggle more than 1 or 2 coprocesses from the same context. You're almost always plugging them together in series or in a complex graph using subshells. In either case you're only needing to manage the same few descriptor constants. It's sort of like writing in a stack-based language, where you really only use explicit references to the top of the stack.

coprocesses provide a simple construct for cases where you would otherwise have to manually create a named pipe using mkfifo. The need for this pattern comes up surprisingly often, IME, yet because it's so messy--now you need to worry about /tmp races, cleaning things up, etc--people rarely bother, myself included. From the perspective of a shell it's a relatively simple construct to implement. The semantic benefit/cost ratio is exceptional.

This basic functionality is complemented with extension options to the read and print (non-standard) builtins: -p reads from or writes to the latest coprocess, and -u the specified descriptor number. I haven't used coprocesses much because I can't rely on them, but I don't think these are strictly needed. Maybe they were required for proper buffering in early implementations (or even current ones), where redirection to and from builtins didn't utilize the same buffered I/O context? That seems more like an implementation detail.

ksh93, bash, and some other shells provide the process substitution construct, "<()", which can provide similar capabilities to coprocesses in many cases. But I'm pretty sure anything you could do with process substitution you could do with coprocesses, making the latter the more powerful construct. coprocesses seems like the critical missing primitive from the traditional Unix shell model, and would round it out. (Of course, the sky is the limit for evolving the Unix shell model in new directions, as Oil has shown.)

ksh88 had coprocesses so I'm curious why it wasn't included in POSIX Maybe because bash lacked an implementation? Or incompatible quirks between ksh88 and ksh93 related to stdout inheritance across coprocesses?

Lack of warnings about accidental file overwrite is less of a problem today than say 20-30 years ago: modern filesystems allows to make regular snapshots which help to recover accidentally lost file. It doesn't happen often in my practice, but by doing what told without questions shell saves my time every day.

Well "mastering [that] careless design" has made me farore capable than many of my peers at tasks that involve debugging network problems. I can slice and dice data files with ease and much more speed than any of my associates because the power the shell brings, I have gained a much deeper understanding of how am OS and computing works, what is going on under the hood, and consequently have reaped many advantages due to many hours spent mastering the shell.

You are right it has some warts, but if a file is really that absolutely critical I make a .bak copy of it before I start my work. The shell has its limitations but so do all tools and the power it gives me is unmatched.

As for other popular software there is plenty out there but other popular software out there destroys something far more valuable to me. My time.

>You are right it has some warts, but if a file is really that absolutely critical I make a .bak copy of it before I start my work

This seems to miss the point completely, unless you back up your entire hard disk before running every command. The whole problem is it's easy to nuke things you weren't intending to touch, accidentally.

The user-hostility of the shell is not a required feature to achieve its power. All of the advantages you cite could be had without being a terrible footgun with human factors mired in 1970s computing culture.

How is any of the advantages you listed not achievable by learning a better programming language, say, Python?

Some of them are but for quick stuff it’s hard to match the speed and flexibility of the shell

The shell is a tool - one of many in your kit. I 100% agree that one should be quick to use Perl or Python for many tasks. But for some tasks, the shell is just the thing to pull out of your back pocket.

The problem I have with this is it’s just software.

I am not a carpenter and context switching is expensive when it comes to thought work (probably because it’s still just typing, lacking the mechanics of swapping a hammer for a screwdriver).

Anyway, the point is by now, 20+ years into an IT career, I should have at least ONE option that can handle these needs without flip flopping through an entire operating systems “tools”.

The desktop metaphor is not the only real world model computing needs to get rid of. I want an interface that can do the job, not a million “tools”.

I'm surprised you didn't mention math. It's astounding to me how fragile it is to add 2 and 2 in the shell. And forget about floating point numbers. Anytime I need basic math in the shell I reach for 'python -c'

It is often easier to reach for my calculator, calculator app, or the address bar, to do arithmetic, than:

      echo "2 + 2" | bc -l

ksh arithmetic includes floating point. Sadly bash made it to the extinguish step before embracing and extending that far. (Compound variables and getopts are the other big omissions.)

Dealing with files and directories? Best option. Dealing with calling multiple other tools written in different languages and/or network calls, and combine, store and do some basic operations on the data? Best option.

It's a job control language after all.

The pipes and redirects, and the instant access to the file system is also very powerful.

Sure it could/should be improved massively, but I'd say it's still the best tool for a significant amount of pretty common use cases, it doesn't just suck at everything.

Try writing this in another language:

  wget http://server{1..5}.com/file.txt >> /tmp/data.txt & wait && wc -l /tmp/data.txt

There's some pretty serious stuff going on there

The problem is we've historically limited ourselves to the lowest common denominator shell so our scripts worked everywhere without modifications.

Hence everything has been stuck with the archaic limitations and oversights of posix shell or gnu bash as linux distros took over and largely standardized on it.

There have been numerous alternative shells and some significantly improve on the situation, unfortunately they've been largely ignored. Default shell choice has proven very sticky.

I do agree that it's practically impossible to write robust and safe scripts in bash/posix shell. You have to jump through many hoops if arbitrary user input is being handled. Frankly nobody should be writing generalized tools using such shells today, unless they have a very good reason to self-impose the constraint.

Yeah, but in the modern era, where almost all the ancient Unixes are for all practical purposes dead (you basically have to be an IBM consultant to use AIX, an HP consultant to use HP-UX, an Oracle consultant to use Solaris, *BSDs probably have 0.01% of Linux' usage rate), we live in a Linux mono culture.

99.99% of everything is internet connected and auto-updates. New distro versions come out every 2 years.

There's no real reason why bash couldn't have been evolved into a sane language with sane parsing and with modern constructs. By sane, I mean... read this:

https://www.oilshell.org/blog/

At least these articles:

https://www.oilshell.org/blog/2021/01/why-a-new-shell.html (especially this one)

https://www.oilshell.org/blog/2016/10/20.html

https://www.oilshell.org/blog/2016/10/28.html

https://www.oilshell.org/blog/2016/11/06.html

If freaking Javascript could "use strict;" and turn a page, bash could have definitely have done the same.

It's just that there's no real commercial interest and FSF doesn't have the manpower (or desire) to do this.

Yes 'use strict;' is a great analogy! I mentioned that here with regard to Perl 5:

https://www.oilshell.org/blog/2020/07/blog-roadmap.html#the-...

It turns out that bash's "shopt" can be co-opted for this, and that's exactly what Oil does. For example, after 'shopt --set parse_paren', you can do this:

    if (x > 0) {  # parens are now Python-like expressions, not subshell!
      echo "$x is positive"
    }

Yes I agree bash could have done this. But the bash codebase is very fragile and things often break. The maintainers are very concerned about breakage.

Oil is written in a way to make breakage less likely: with principled parsing and a detailed code representation using algebraic data types. I'm surprised at how much language evolution this style has enabled. It makes me optimistic about the project's future!

Although we still tons of work to do with the C++ translation -- writing a shell is hard either way!

> Yes I agree bash could have done this. But the bash codebase is very fragile and things often break. The maintainers are very concerned about breakage.

True, but that's not a good reason for such a used application. It has been around for 40 years, I think it literally has trillions of runtime hours. It's used on billions of systems, most likely.

It's exactly the place where you'd create a super comprehensive test suite that covers <<everything>> (see SQLite) and then re-architect it to make it robust.

I wonder if it's due to lack or money or due to something else.

> There's no real reason why bash couldn't have been evolved into a sane language with sane parsing and with modern constructs. By sane, I mean... read this:

> https://www.oilshell.org/blog/

I think the point is that, it did evolve, as your link shows—and there are lots of other sane alternative shells, too. The problem isn't that they don't exist; it's getting people to adopt them in sufficient numbers.

> *BSDs probably have 0.01% of Linux' usage rate

Just wanted to point out that Hacker News runs on FreeBSD[0] :) Some people like the BSDs, and would appreciate it if everybody stopped designing for the Linux monoculture (well, monoculture if you disregard Windows and macOS).

[0] https://docs.freebsd.org/en/books/handbook/introduction/

Hacker News runs on FreeBSD and Arc, Arc being a super obscure Lisp dialect that I can't imagine being used in production anywhere else (for sure not in a place with a similar visibility or impact).

"What HN uses" doesn't say much about tech trends :-)

Edit: You probably want to use Netflix as a better example. I think it's used by their CDN layer.

Don't forget filenames that start with a dash... Usually you need to remember to put "--" in every single command invocation.

That has to do absolute nothing with the shell, that's a Unix convention how commands parse their arguments.

I disagree, the shell is in the best situation to do something about it. Or at least it would be, if there was a consistent way for programs to report to the shell how their argument parsing works.

that would be horrific

More horrific than having to guess what switches are supported by any given command on the target machine, even though shell scripts are supposed to be cross platform?

Yes, those are all problems with bash and the state of the art.

The funny thing is that there about a dozen ways to spell it, like $(dirname $(readlink -f $0)). Since the shell itself doesn't provide this functionality, people come up with a lot of workarounds.

In Oil it will simply by $_this_dir, e.g. "source $_this_dir/mylib.sh" for relative imports. (Whenever a variable is silently mutated by the interpreter, akin to $?, Oil prefixes it with _).

https://github.com/oilshell/oil/issues/587

I solicit feedback on every Oil release: http://www.oilshell.org/blog/tags.html?tag=oil-release#oil-r...

so let me know what you think of the fixes: https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback

Dealing with filenames is fixed in Oil: http://www.oilshell.org/blog/2021/04/simple-word-eval.html and with QSN:

http://www.oilshell.org/blog/2020/10/osh-features.html#safe-...

Again, if you don't think this actually solves the problem, leave some feedback. Oil is the most bash-compatible shell by a mile, so you might need to use it someday :)

(The theory is that in MOST situations you don't get to choose your language, just like you didn't choose C, C++, or shell. It's all the inertia of compatibility.)

Oil has named parameters: http://www.oilshell.org/release/latest/doc/idioms.html#use-n...

I know about the long file list problem, but I don't think I've ever run into it, mainly because I use xargs, which batches up commands correctly. If there is some other situation where it comes up, I'm interested.

Insane combination of grep / awk / cut: Oil has eggex which compiles to ERE to help you write egrep and awk patterns: https://www.oilshell.org/release/0.8.11/doc/eggex.html

You get syntax errors when you write the pattern (at parse time), not when you run it.

Crucially, we don't "rewrite" grep and awk. We just make them easier to use (optionally). There is a smooth upgrade path and you can retain your knowledge while forgetting about some sharp edges.

> Personally I find it amazing that people invented this thing, then spent decades tripping on the same quoting and filename handling issues and not finding that to be a good reason to fix it.

Yes I quoted David Korn complaining about the quoting problem in the early 90's, which was closer to Unix's invention than we are to the early 90's.

https://www.oilshell.org/blog/2019/01/18.html#slogans-to-exp...

The reason we still have it is the inertia of compatibility, and the fact that nobody really owns shell. Even though people rightly complain about Google's web stewardship, HTML5 was a great improvement and cleanup. They paid people to fix HTML, and it worked to a large extent.

Some food for thought: A Generation Lost in the Bazaar https://queue.acm.org/detail.cfm?id=2349257

Maybe you should use a line of Rust in Oil so that the rewrite-it-in-Rust people make Osh into the new ubiquitous shell.

There is probably some use for Rust indirectly, as there are several use cases for WASM in shell, and Rust has some unique advantages for writing WASM:

https://github.com/oilshell/oil/issues/941

(This probably won't happen for a long time though, unless someone really wants to contribute and own it!)

You might want to have a look at the fish shell or oil shell, some progress has been made

> These days if it's more than 5 lines of code, or if arbitrary filenames are involved I go straight to Perl or Python

Totally agreed. A shell script exceeding that's about 10 lines of code becomes so complex to read, interpret and improve that even though it may perform faster I would rather get the job done in a language that I can understand better when I come back to this 3 months later.

Although ... on that note I don't think I'd ever choose Perl ha ha.

One does not... simply choose perl.

> How do you find where the script itself is?

This question does not really make sense. There need not be any script, or there may be many at the same time. What exactly do you want? Imagine, for example, that you are piping the output of some commands into the shell.

>There need not be any script

Sure, but that is irrelevant.

When there IS a script, you should be able to get its full path much easier, that is the complaint.

I agree that it may be possible to implement this feature in a nice form where it makes sense (for the people who need it). But it seems really strange to me and I cannot imagine a use for it. Conceptually, the shell only sees the contents of the script, why would it act differently according to where the script is? When you open a jpeg image, the jpeg library does not provide a facility to recover the directory where that file was found. For all you know, this may be an in-memory jpeg. Isn't the same with shell scripts?

>Conceptually, the shell only sees the contents of the script, why would it act differently according to where the script is?

Doesn't need to act differently in what it does.

But it does need to report it, because e.g. you might want to bundle data/assets/config relative to the script, and want to be able to run it from wharever and have it still be able to find it.

It's an interesting question. In most shell, it is possible to "source" a given file, which basically is a form of file inclusion. How should the script location feature behave in that situation? Should it report the sourced file location, or the sourcing one?

In the later case, does it mean that it is not possible to write reusable code based on that feature? In the former case, what should be done when sourcing is done recursively? Can this be used to defeat file location, for instance, by generating a script in /tmp and sourcing another file from it?

First of all: https://en.wikipedia.org/wiki/Argument_from_incredulity

Secondly, it's super frequent that scripts I write or that others write need the folder they're executed in as an argument.

99% of scripts I've seen aren't meant to be piped to. They're just launched as simple, dumb, standalone "programs".

In that case, of course, the method of finding where the script lives would return something that indicates there is no script. But also in that case, the commands you are piping into the shell probably are not looking for the files they were bundled with, they are probably working on files you've explicitly specified, and so there is no problem with not finding the script's location in that case.

This makes me really hope to see more uptake of PowerShell in Linux (and performance improvements in PowerShell too). The above is just $PSScriptRoot.

as I comment in basically every one of these shell articles, while Unix shell does kinda suck, the main problem is that the bad examples far, far outnumber the good ones, and so everybody programs in an overly complicated manner.

  SCRIPT_DIR="${0%/*}"

works in practically all shells, as long as the script is actually invoked by path (if you do curl script | bash, you're SOL right off the bat).

the same applies with "combination of grep, awk and cut". awk contains almost all the functionality of grep, sed, and cut, so any pipeline of awk plus one of the others is almost always unnecessarily complicated. it would be like complaining "in python, string handling is so hard. look, you have to do import re; for x in re.sub(...).match(5).split(' ')[3:-5]: lst.append(x)". of course if you make it overly complicated then it will be overly complicated.

there are also too many footguns. spaces and newlines are trivial to handle in 95% of scripts, as long as you quote every expansion. as long as you never ever write cmd $file and always write cmd "$file", that solves virtually all whitespace problems. quotation marks in filenames are never interpreted in a special manner in Bourne-like shell unless you use eval, which has the same pitfalls as any other interpreted language (see python above).

Now, invoke a script, containing that code, in the current directory, and you will see why his example was more complex.

    SCRIPT_DIR="$(dirname $0)"

It also works when you're not running bash.

This doesn't work if $0 contains whitespace. Fix it by using "$0" instead.

LOL. What kind of savage runs a script with whitespace in its name?

What kind of savage OS doesn't even have a standard text encoding for its filenames and even allows its filenames to be binary?

Just because we wish something weren't so, doesn't make it as we wish it.

That's essentially what he did, but he also accounted for the case of running a script from the current directory by running pwd.

> How do you find where the script itself is?

No program can do this portably.

I think you might be mixing two things up: it's impossible for a native program to find where it is, because argv[0] can be anything.

But it's possible for an interpreter to tell the script it's interpreting where it lives, i.e. argv[1]. For example, Python does this in a pretty straightforward way.

You're right, my bad.