Hacker News Reader

Comment by chubot | Hacker News Reader

(author here) Hm I've tried fish, and it seems very nice for interactive use.

However I don't see it being used AT ALL for the cloud/linux use case? Those are the cases where you tend to get 1000+ lines of shell scripts.

For example, I mention Kubernetes/Docker/Chef, and I've never seen fish in that space.

I also don't know of any Linux distro that uses fish as their foundation -- they all appear to use a POSIX shell like bash/dash/busybox ash. fish is "on top".

See Success with Aboriginal, Alpine, and Debian http://www.oilshell.org/blog/2018/01/15.html -- these distros are built with thousands of lines of shell scripts.

Either 1) I don't know about such usage, 2) people don't know that fish can be used this way, or 3) there is some problem with using fish this way.

I link to this post in the FAQ, which I think is a lot closer to Oil:

https://ilya-sher.org/2017/07/07/why-next-generation-shell/

It's basically the "devops" use case. (And as I mention the main difference between Oil and NGS is that Oil is compatible / has an upgrade path from bash.)

View on HN

It's an entrenched network effect. There's a lot of existing scripts written targeting POSIX shells, and there's an incentive to have all your distro maintenance scripts written in one language if possible.

If you want to switch your existing distro scripts to fish, you need to rewrite a lot of stuff. If you want to start a new distro, "it's written in fish!" isn't a terribly compelling selling point, since I don't think anyone is actively picking distros based on their tooling language.

For the general devops case, there's a lot of existing example code out there for bash scripts, and far less for fish. "How do I do X in bash?" is probably going to get you a decent example on Stack Overflow. Devops (I feel) cares a bit more about the installed-everywhere thing, and fish isn't a standard part of most distros.

Oil having a goal of being completely sh/bash compatible gives it much greater odds of something like Debian switching to it, since it wouldn't carry huge technical debt along with it.

> I don't think anyone is actively picking distros based on their tooling language.

I would use the crap out of a Linux distribution built in python.

yeah I can actually imagine that. like using an IDE based on the language it was written in to extend and tweak it more easily, or a window manager etc..

actually now you mentioned it, barring possible performance issues I'd also like a Python distro

> an IDE based on the language it was written in

I recommend Emacs. Maybe not as a daily driver (it's a matter of preference), but for the experience. I recommend at least a month with it, make sure to write some original Lisp code for your customisations. (You WILL end up customizing it, the defaults are crap.)

> or a window manager

I recommend Awesome. The core is in C, but that's basically the low-level stuff, the actual WM is written in Lua. If you'd start with an empty init file, you'd have a moderately sized side-project on your hands to put something usable together. But if you'd start from the stock rc, there's a world of endless tweaking and customisation waiting for you...

Watch out, both Emacs and Awesome are rabbit holes.

Wait, what? You recommend Emacs but not StumpWM?

I've used -- for longer periods of time, of course -- Awesome, Xmonad, i3, and dwm; while some of them are better at some things than StumpWM, StumpWM is the only one that provides an Emacs, live-hackable experience. So these days, I strongly prefer that.

I've been hearing about Emacs X Window Manager but haven't given it a shot yet https://github.com/ch11ng/exwm

> the defaults are crap

As someone who has recently joined the 1k LOC Emacs configuration file club, I have to disagree. The default settings are a well thought out starting point that needs minimal tweaking to get to exactly where you want. I try out a lot of packages on ELPA, and many that purport to provide alternatives for defaults end up being inferior to using the defaults with some small tweaks. The convenient thing is that you can customize the third-party packages in the same way and just use the parts you like from them.

The other good thing about Emacs is that you don't need a separate window manager[1], and you don't need to use a separate command shell (Emacs comes with eshell), or to use a terminal emulator for SSHing into remote machines (eshell over TRAMP is absolutely amazing), or a file manager (Emacs comes with dired, which works over TRAMP which is amazing), or a...

[1] https://github.com/ch11ng/exwm

eshell over TRAMP? how does that work?

Any pathname in eshell can be a TRAMP pathname on a remote machine. So you just cd to a remote directory and eshell will run commands on the remote machine. You can redirect remote command output to a local file, local command output to a remote file, cp from remote pathname on machine A to remote pathname on machine B, etc.

My favourite Emacs goodie to highlight is magit because it really changed the way I use git. All of a sudden, it's incredibly simple to (un-)stage individual hunks, etc. And of course it works over TRAMP, too :)

org-mode is a good one, too, but I feel like I've barely scratched the surface there.

The hunk feature of magit is great; you can easily stage/unstage not only at the hunk granularity but also arbitrary lines using the selection. Being able to easily do that was something I really missed when I moved from darcs to git.

When it comes to magit and TRAMP, definitely the best part is with-editor[1]. This is one of those ingenious "why didn't I think of this?" hacks to let you use your local Emacs as EDITOR on remote machines without having to do SSH port forwarding for emacsclient.

[1] https://github.com/magit/with-editor

At some point I realized that I wasted months in those rabbit holes customizing and tweaking shell configs, window managers and Emacs. Any gain in productivity (which may not even be there as it is rather subjective) was not able to offset it. So I stopped it. These days I just use Gnome Shell with a single extension to fix something that I cannot get used at all, simple bash config to setup environment for development with no customization on shell behavior, and a programming editor that mostly matches my taste. On a new machine I can finish all customization within 5 minutes and forget about it.

I even started to suspect that an easy customization of a tool can be a bad sign. It is almost like developers delegates all usability issues to the end user making the default case really bad.

https://news.ycombinator.com/item?id=16152639 :)

Early Smalltalk and Lisp systems were those langs all the way down. Using emacs with Common Lisp or using Pharo Smalltalk can give you a feel for this, but not exactly the same thing as a Symbolics lisp machine or the Xerox Alto. Alan Kay has a presentation where he shows the entire OS, word processor, networking stuff, IDE, paint programs...etc was an insanely small amount of code and everything was user configurable. I really wish that had caught on more. I really enjoy Linux (although it ain't perfect). Using Windows and trying to interactively do anything with the OS is an exercise in pain and frustration. If someone could build a modern machine with an OS that is like that and had a few basic apps (web browser...etc), I would be quite happy.

I don't know if it's just me, but I don't feel like I'm using a computer when using Windows. After using OpenBSD exclusively for weeks (my laptop broke, so I had to repurpose an old netbook with OpenBSD), I felt bored and limited in Windows. I tried Powershell, but -mostly because I don't know much about it- it didn't give the same feeling.

The thing that is working for me is to explore VB.net . It is really an interesting language with many historical features. I've noticed that all the languages I use (Python, C#, Java, Go, etc.) distracted me from the essence of programming. When I started programming I immidiately thought about patterns, architecture, unit-tests, integration-test etc.

In VB.net I feel like I'm slowly reexperiencing the joy I've felt in my earlier days of programming.

you can turn windows into an adult computer by using cygwin (maybe mingw), autohotkey and something like window manager.

the illusion is almost real, because sometimes i also run linux in vmware/docker/... and when i am not paying attention i forget easily which os i am on.

the unix subsystem for win is quite boring as you can't mix windows applications with linux ones, for example running visual studio build from bash.

powershell would be probably worth learning, if i'd have to, but i do not. it may be godsend to win sysadmins though, as classic cmd is a 40 year old joke.

You can run windows programs out of bash (I mean the wsl bash), and you can also create wrapper bat files to use Linux tools/programs from Windows.

i see ( https://docs.microsoft.com/en-us/windows/wsl/interop )

that was not possible when i tried it, thanks for pointing it out

True, but it's a hack...not something fluid/elegant/seamless or efficient. I'm not a software engineer. I'm just a traditional engineer that wants an OS written for a power user that allows me to do my job more efficiently. Dealing with a multitude of wrappers is worth it if you're writing production software of course, but not if you write a lot of one off scripts for day to day work.

"In VB.net I feel like I'm slowly reexperiencing the joy I've felt in my earlier days of programming."

Ruby was the first language that I experienced that got out of the way, so I felt like I could think about what I was trying to achieve, rather being distracted by house-keeping. Of the other languages that I've tried, I've only gotten close to the same feeling with JavaScript, Go and Python.

I suspect that a lot of it is to do with your own mind-set at the time: I first learned Python as an applications language, with all of the add-on tools and baggage, and didn't like it much, but recently I wrote a one-file, no-dependency thing to do exactly one job in Python, using just the standard library, and it was a joy.

The difference was, I think, that I did not feel the internal pressure to meet best practices, and could just write the code. For some languages, the stuff that you are supposed to do and the tools that you are supposed to use are a heavy burden.

I've stated on here numerous times that Powershell falls short because pipes aren't text based as in Unix, but are really more like method chaining in .NET. Most of the time I have the output of one command, but can't easily reason about passing it to the next command.

I feel you. This guy is working on an interesting project called the Reform laptop [1]. You might have seen this already. He will eventually have it run his own lisp (called "Interim") all the way down.

[1] http://mntmn.com/reform/

Sounds neat, I'll check this out.

IIRC, Python performance is better than than bash.

Not in my tests involving repeated invocation of binaries, iirc.

Gentoo is basically this. I learned Python by writing .ebuilds (their package format is a short Python script that downloads and builds whatever software you're installing).

Although emerge is written in python, Afaik ebuilds are bash scripts.

Ah right you are, thanks - I remember learning Python and switching to Gentoo about the same time, but as David Hume warned us, it's hard to infer causality from proximity.

It could happen:

https://github.com/dcantrell/pycoreutils https://github.com/davidfischer/pycoreutils

http://u-root.tk/ does that for Go.

Not Python, but GuixSD is built around (Guile) Scheme, from the package manager (Guix) to the init system (shepherd).

Pardus existed. Many distros have bits of python used here and there.

The fact that there is the POSIX standard and multiple shells that meet that standard is also a big part of it. If you write in a POSIX compatible shell language you know it can run anywhere. Fish, for example, has no standard and only 1 implementation.

Any new shell that hopes to compete in the systems space will have to be POSIX compliant. At least until there is a new widely adopted shell standard.

Back when I started programming, I wrote 1000+ lines of shell scripts.

Now, I quite seriously believe that a 1000 line shell script only exists out of error. I still occasionally end up doing 2-300 line dense shell scripts, but not without feeling very dirty along the way. Either split into small, simple shell scripts (which is fine), or a different language.

In the cross platform build pipeline at work, I keep a strong discipline when it comes to scripts: They must be short (<=100 lines), and if their complexity exceeds a certain threshold (parsing more than a "grep | cut" here and there, total program states exceeding some low number), then a shell script is no longer acceptable regardless of length. And, well, it's not safe to assume the presence of anything more than a shell.

If you are writing and dealing with 1000+ lines of shell scripts, then experience tells me that you are shooting yourself in the foot. With a gatling gun.

(I used fish, btw. The interactive experience was nice, but the syntax just felt different without much gain, which was frustrating to someone who often writes inline oneliners. Unlearning bash-isms is not a liberty I can afford, as I need to be proficient when I SSH into a machine I do not own or control. I can't force the entire company to install fish on all our lab servers, nor is it okay to install a shell on another persons' dev machine just because we need to coorperate.)

If your script is over 80-120 lines, it's time for modularization. No sane person writes 1000 line scripts, and the fact that the Oil author uses such a scenario as a reason to use their shell is very disconcerting about the quality of code and I wouldn't touch Oil with a ten-foot pole.

If that's the case, then there are a lot of insane people out there. I sympathize with your opinion on how things should be (though I may or may not agree), but at the end of the day we need to look at what the reality of the world is, not on the ideal way we think people should be using the tools we already have.

Well to be fair, there are a lot of insane people out there. :-)

Just because writing 1000+ line bash scripts is something that happens doesn't mean it is something one should not avoid.

People also implement new SQL injection bugs every day—just because a lot of people do it doesn't mean we should just let it slide.

Please read the FAQ:

http://www.oilshell.org/blog/2018/01/28.html#toc_7

I didn't write those scripts, and you rely on them, whether you know it or not.

Do you use Unix? Do you use the cloud? You rely on them. See:

https://www.reddit.com/r/linux/comments/7lsajn/oil_shell_03_...

Also, shell has functions.

I see. My apologies for being so critical. I agree that POSIX is incomplete and out of date.

However I do think it is a disaster that people can't break up their shell code into more modular components, and there is no excuse for it.

And I especially do not wish to discourage you from trying to reinvent shell, because someone has to do it, but at minimum I don't see myself moving away from my shell for the better part of a decade. Hopefully at that point Oil's community has matured to the point where I feel comfortable trusting it to be stable and free of any critical privilege escalation bugs.

Good luck!

I agree that the age of 1000+ lines of shell script code should be over; just replace the shebang and use whatever other language you like instead. Unless you need something dependency-free, compilation-free, and portable, then God help you because sh it is.

It's funny though, after decades of shying away from them I've now gone full circle and embraced Makefiles once more. Perhaps it's the cleanness of the bmake/pmake extensions as compared to those of GNU Make, but the determinism, zero dependencies (no CMake or meson and its python baggage), automatic parallelization, and strict error handling call to me each time I have to write mission critical "glue code" that must. just. work.

I would always pick a make file over a script when it's appropriate - why did you shy away from them?

Probably because of how much misinformation there is about them and how prevalent bad Makefile code is, both in tutorials and in the real world. They're so fundamentally simple - it's just a directed graph - that they're easy to get wrong (mainly because they might still work.. until you need to change something). The majority of Makefiles I encounter will quickly break down under high levels of parallelization due to mistakes in the DAG (try -j64 on random projects from "huge" "high-quality" open source projects). Add to that the fact that most projects use autoconf to generate Makefiles when they don't need to, it was just a mess.

After learning and using many other "modern" Makefile replacements of various complexities (ninja, cmake, scons, tup, meson, bazel, and others) I realized if you're not using exactly what the tool was designed for, you end up recreating a Makefile (e.g.: meson is awesome for cross-platform C++ builds, but if you try to use it to build an environment composed of output from various processes that aren't C++ compilers, writing a Makefile is easier). CMake, apart from also being too C/C++-specific, would be nice if it didn't require a million different files and didn't have such a god awful language.

The only one I liked as a general purpose build system was ninja but it is too restrictive to code in (requires duplication of code by design, not meant to be written by hand though I still do from time to time) and tup, but tup is built around fuse (instead of kqueue/inotify/FindFirstChangeNotification) and so is a no-go for anything serious.

Once I embraced Makefiles, it turned out that most things traditionally built with shell scripts should actually be Makefiles for determinism. For example, I just used bmake to take a clean FreeBSD AMI and turn it into the environment I need to perform some task every n intervals (the task itself was turned to a Makefile rule), in place of where Puppet and other tools would normally have been needed, but would have been overkill for my needs.

The only drawback to Makefiles that I haven't found a clean solution to is when you need to maintain state within a rule (without poisoning the global environment). The only solutions I can see are either a) using a file to store state instead of a variable, which is just stupid, b) calling a separate shell script (with `set -e; set -x` to try and mimic Make behavior) which sort of defeats the point of Make, and c) multiline rules with ;\ everywhere, which is hideous and error-prone but works (though it makes debugging a nightmare as the rule executes as one command, again defeating some of the benefits of Make).

> dependency-free, compilation-free, and portable, then God help you because sh it is.

That's a fair point. Nothing beats shells for ubiquity and supportedness everywhere . . . in theory.

In practice, even people who write "portable shell scripts" (almost) never really write portable shell scripts.

If you learn the POSIX sh standard like the back of your hand, and stick only to the features in it, never using bashisms or equivalent, most shellscripts still rely on external programs (even if only grep/sed/etc) to do their heavy lifting.

And that's where you get into trouble. Because compared to the variability in behavior of even "standard" ultra-common programs, the variability in behaviors between shell syntaxes/POSIX-vs-non-POSIX shells is tiny. The instant your code invokes an external program, no matter how ubiquitous that program is, you have to worry about:

- People screwing with PATH and changing the program you get.

- People screwing with variables that affect the external program's behavior: LD_LIBRARY_PATH for dynamic executables, or language-specific globals for programs' runtime behavior (CLASSPATH, PERL5LIB, PYTHONPATH, etc.).

- External program "editions" (e.g. non-GNU sed vs GNU sed). Good luck using PCRE with a non-GNU "grep"! Oh, and if you're sticking to POSIX-only shell semantics (no bashisms), you don't even get full BREs in most places you need them in the shell; you're stuck with limited BRE or globbing, which makes editing 100-line PCRE regexes feel like a dream.

- External program per-version differences.

- External program configuration.

- Etc.

Dealing with those issues is where "self-test"/"commandline program feature probe" things like autotools really shine. Raw shellscripts, though, very seldom live up to the "ubiquity" promise.

> I quite seriously believe that a 1000 line shell script only exists out of error.

Perhaps the best use case for Oil is to provide a debugging environment where you can figure out what your legacy shell scripts are doing, and rewrite them in another language.

> Unlearning bash-isms is not a liberty I can afford, as I need to be proficient when I SSH into a machine I do not own or control.

This is a slippery slope. I have heard things like, "don't make your own custom aliases / shell functions, because they won't be available when you SSH to another machine." Forcing yourself to always use the lowest common denominator of software is not a fun path.

Yes, absolutely. Oil has a principled representation of the interpreter state, so it should be easy to write a debugger.

(Although note that bash actually has a debugger called bashdb, which I didn't know about until recently, and I've never heard of anyone using it.)

One intermediate step I would like to take is to provide a hook to dump the interpreter state on an error (set -e). Sort of like a "stack trace on steroids".

If anyone is running shell cron jobs in the cloud and would like this, please contact me at andy@oilshell.org.

I want to give people a reason to use OSH -- right now there is really no reason to use it, since it is doing things bash already does. But I think a stack trace + interpreter dump in the cloud (like Sentry and all those services) would be a compelling feature. I'm looking for feedback on that.

Shell scripts have a very small amount of in-process state, so it should be easy to dump every variable in the program.

Also some kind of logging hook might be interesting for cloud use cases.

> Forcing yourself to always use the lowest common denominator of software is not a fun path.

Second this. Been there, and back, and there again, and recently back again.

Customize the hell out of your shell, make it your place, make it nice. You're spending your day there, every day. Treat it like you'd treat your work desk.

If it's a stranger's machine, well, OK, suffer through the 10 minutes of troubleshooting, an occasional session with busybox also helps keep "pure POSIX" skillset fresh. If you're becoming a regular there, it's time to think how to make your dotfiles portable. google.com/search?q=dotfiles

While I appreciate your perspective, I think it’s more from the “occasional SSH to a foreign server” mindset. Scaling “the way I work in a shell” to huge fleets is a nonstarter, and as a working SRE, has been the biggest thing holding me back from zsh, fish, and other alternatives.

OTOH, I’m also against, for example, installing zsh fleetwide in production fleets to accommodate choosy folks. So I’m on both ends of the problem, and know it.

It's actually not that difficult to maintain context if you use completely different shells (with different-looking prompts) on your own machine vs the fleet. It's like code-switching when you talk to your friends at the bar vs co-workers.

- As soon as you have more than 5 boxes, you need different tools for your everyday tasks, interactive SSH sessions just don't work. But you know that already.

- So you've found yourself SSH'ing into a particular box, because that's just the best way to work on a problem. Either it's a pet box (in which case it makes total sense to drop your dotfiles there), or it's a cattle box and you don't care about the trash you left behind, because autoscaling will clean it up for you.

My zsh/nvim/... dotfiles are portable and compatible with FreeBSD, macOS and a bunch of Linux distributions, but it's absolutely unacceptable to run those on a machine I don't control. You don't mess with configuration on another guy(ette)'s server.

My setup is customized, but the core experience is kept relatively untouched. If someone experiences a problem, I need to be able to debug it in place efficiently. I write drivers for the company's hardware for a living—troubleshooting takes more than 10 minutes.

> Perhaps the best use case for Oil is to provide a debugging environment where you can figure out what your legacy shell scripts are doing, and rewrite them in another language.

My approach to that problem is to just instrument scripts to write out the command line every time a tool is called, and infer the logic behind it. When I started here, build/package/test on Windows were controlled by 6000 lines of Perl... A printing version of system() and a few days later, it's 40 lines of batch and 150 lines of Python.

> Forcing yourself to always use the lowest common denominator of software is not a fun path.

Never said it was fun. Also, I need to be able to use the lowest common denominator, I don't necessarily have to be the lowest common denominator. I use a configured zsh and nvim instance myself, but I take care to ensure that if I'm stuck with vi and (ba|da|k|c)sh, I'm still productive. The core behavior of my VIM instance stays close to stock, but it has a bunch of extras like fuzzy search, linting, a nice theme, etc.

And, if I can make a choice without caring, my needs for a shell and a programming language are in direct opposition. I want my shell to put all of its energy into a useful interactive experience.

> "don't make your own custom aliases / shell functions, because they won't be available when you SSH to another machine."

If you're logging into a machine to the point you're editing files or require your customizations in this day and age you're doing it wrong IMHO.

Doing what wrong?

And what would you recommend instead, then?

Bash is flat-out not a scripting language. It is a command language. It does not support typed variables, or named parameters, or any number of basic scripting language features. It technically does not even provide an 'if' construct. It's sufficient as a 'glue layer', and the Unix toolchain is nice, but it provides next to nothing in the way of abstraction, and that's liable to become a problem closer to the 1000 character mark than 1000 lines.

My rough heuristic is, "no more than ten lines, nor more than two variables." Yes, that's short almost to the point of absurdity. The only good thing that one can say about Bash as a scripting language is that it's better than csh. Bash is taken seriously because of its longevity and ubiquity, but it's fundamentally limited in what it can express, and it is quite trivial to exceed those limitations.

This one gets it. What I want from an interactive shell is not what I want from a programming language.

I wondered this while reading the article. Do we really need the same language for commands & for programming? Portability is nice, but it seems like the requirements are a bit different for one-liners vs scripts you need to maintain.

Well, Perl back in the bad old days (4 and earlier) was very similar in terms of limited "minimum standard" facilities, but tons of people used it as a general purpose scripting language and hacked together some really huge systems.

It's definitely easier/safer to write longer programs in other languages, but with proper discipline it's also possible to write really big, robust programs in Bash/shells--people have done this, and some of that code is probably still running today.

> and some of that code is probably still running today.

A lot of that code is still being written today, which is one of the things that OP is trying to change.

> They must be short (<=100 lines), and if their complexity exceeds a certain threshold then a shell script is no longer acceptable regardless of length.

Well, the question is: what do you use as a replacement? For my scripts I tend to pull in some version of PHP as soon as needed. Perl is something that's universally available in anything based on Debian or Ubuntu, but it's a maintenance nightmare, and PHP doesn't care whether I use spaces, tabs or a mixture, or if there are due to any circumstances mixed line endings in a file whereas Python may or may not barf on encountering any of said things...

Anything that supports pipes and signals, and is easy to install. I've started using Steel Bank Common Lisp (SBCL) because it's the default implementation used with the Roswell installer.

https://github.com/roswell/roswell

(As a side-note: Perl 5 was really the ideal language for this, by design; we might be in a different world today if it hadn't been derailed by Perl Forever^W 6).

I mean, Perl 5 wasn't actually ideal for this, but it was better suited to the task than anything else that actually exists.

Perl 6 actually exists and has a REPL which Perl 5 lacked.

If I was ever to be religious, I would believe that PHP was the work of the devil. It's just so... uuuugh. It doesn't make sense. Did you know that you can increment a string in PHP? Yup. Of course you can't decrement it, that's crazy talk.

Anyway, I usually avoid perl. While definitely a true programming language, I really don't like it. I find that it lacks clarity, and quickly collapses into bash-like hackery.

I personally use Python when it's a notch above shell, but not enough to pull out the big guns.

(I use to write a lot of perl when I was a kid, even made an accounting system in it, so my dislike of perl is from experience. We also had a 6000 line perl build script at my current job which was the last drop. Rewrote it to 40 lines of batch and 150 lines of Python. Likewise, I also wrote a lot of PHP, and can no longer consider it a proper programming language.)

It sounds like the original build script was poorly written. Bad (or hurried) programmers write bad code no matter the language.

Are you claiming that its size was because Perl forced it to be long winded?

My dislike of Perl comes from personal experience in using it. I don't think one can have valid strong opinions of a language without having used extensively.

I feel like some programming languages, no matter the structure of the application, feels unstructured and write-only in nature. To me, Perl is one of such languages. Python has a similar effect, but I feel like it is more controllable as long as the project is kept within a sensible length.

So while that abomination was primarily the fault of the designer, I do think that Perl invites this behavior one way or another.

(I gave up entirely on reading that disaster of a build script, and instead instrumented system() so that I could trace the calls to various tools and infer the logic myself.)

My personal threshold for upgrading from shell to Python is about "I need an if statement".

Yeah I've seen that Python code. Every other line is a call to os.system('...').

You say that like it's a bad thing. Bash does not support niceties like typed variables, named parameters, or classes. Its conditional operator is an external program with a required last argument to make it look like sane syntax. Little things like interpolating variables, array access, or taking command line options are awkward and prone to failure. It doesn't have the insane syntactical issues that csh had, and POSIX is ubiquitous, but that's a pretty low bar and an inadequate justification for using a fundamentally limited language. Python could have a nicer syntax for invoking the system interpreter -- personally, I think that Ruby's backticks feel very Unixy -- but if there is a better interpreter on the system there is no reason not to use it. Even if half your code is calling out to the shell, you still have the benefits of typed variables, a nicer syntax, the ability to abstract code into classes, a variety of nice enumeration options, and all the other benefits of a "real" programming language.

declare -a myArray=("foo" "bar" "baz")

echo "${myArray[2]}"

Bash is weird.

I've had good luck with this module. Much nicer than using os.system() imo.

https://amoffat.github.io/sh/

never use os.system. os.system is a hack.

Subprocess is a bit verbose to use, though... A wrapper would be nice.

Since python3.6 it's just subprocess.run(). The SP module was refactored. I hope that eliminates the need for all these annoying (IMHO) wrappers.

Ah, that's nice!

... unfortunately, I'm stuck with python2.4 or 2.6 on most machines my Python code has to run on. :/

> I'm stuck with python2.4 or 2.6

Not even 2.7?! 2.6 hasn't had a security update since 2013[0], I dread to think how old 2.4 is.

[0]: https://www.python.org/download/releases/2.6.9/

Python2.4 is what RHEL/CentOS 5 has to work with... And the extended support for that bloody release lasts till 2021.

You can install a newer version, but our CentOS 5 machines are meant for compatibility tests, so modifying their setup is unacceptable.

Have you tried using the scl repos from RedHat/CentOS? I use it at work yo enable different versions of Python at a system level if I am unable to muck with it because of yum.

I believe they are available for CentOS 5.x?

I wasn't recommending os.system() ever, OP used it in the example. Subprocess is better, as you said. I think the sh module is better than Subprocess as well.

Every other line of my code is check_call. Could you expand on the issue you're observing?

check_call for the win! I contributed check_call to the subprocess module for exactly that purpose - easier shell like scripting in python :-)

> consider these two groups of shell users:

> 1. People who use shell to type a few commands here and there.

> 2. People who write scripts, which may get into the hundreds or even thousands of lines.

> Oil is aimed at group 2. If you're in group 1, there's admittedly no reason to use it right now.

From my perspective, fish is basically the opposite of oil in that it mainly targets group 1 (admittedly I'm biased in that I rarely write shell scripts at all). IME fish scripts as a direct replacement for bash scripts are pretty rare, but that doesn't mean it's not a success if it's aimed at interactive use.

"to type a few commands here and there"

Where's "my shell is my desktop environment"? :/

/pedantic

Everyone's desktop environment is a shell.

But any shell you like is suitable for that use case since it doesn't have to be compatible with anything other than your fingers.

First, I think it's a case of momentum. All the scripts are written in shells that are distributed by default in most distros. sh, bash, etc. I write lots of scripts for fish, and I can't convince my co-workers to install fish to run my scripts. They already have bash, they already know bash, etc.

Second, I was recently chided by another fish user for writing fish scripts: "you're using it wrong" was the sentiment. The argument was it's a shell intended to be interactive, but extendable. I don't know how pervasive this line of thinking is, but who's going to learn all about their shell if they're not supposed to write scripts in its language?

I've been using fish for years but I never write scripts in it. What's the point?

There's sh which is the most portable. Why write scripts in anything else but an actual programming language when things get more complex?

The point is the compose-save-edit cycle: Compose the script in the shell, save it to a file, and then, maybe, edit it to add better argument handling or support for corner cases you didn't consider in the interactive environment.

One question though:

Do you really need 1000+ lines of shell scripts? I almost have 20 years with shell scripting and I still have the rule of not writing longer than 1-2 page shell scripts and use something else for longer projects or break it down to much smaller chunks and invoke the parts with a main.sh.

Fish is a for making software engineering and system administration easier by having functionality like command completion (without tab) and it works like a charm. I would never ever would nother to rewrite my bash scripts in fish simply because bash is on all the servers and fish is on none by default.

I am not sure what you are trying to solve here but good luck.

See https://news.ycombinator.com/item?id=16278141

This is not an answer just a problem statement.

Totally off topic: great blog. I was reading the archives these days, they’re awesome ;)

Thanks! Feel free to comment (with the reddit links at the bottom of each post)

My two main comments would be:

1. Keep up the great work (both oil and the blog!) Both seem fun :)

2. Reading your blog really shows that despite the great ideas, shell languages come from an age where computing was still in its infancy. We’re now approaching our teen years, but we’re still ahead of infants. So many awkward design decisions and organic evolutions. A shame that replacing POSIX at this point is a gargantuan undertaking.

What is your take on Ion? https://doc.redox-os.org/ion-manual

Ion looks great, but its goals are different -- most importantly, it doesn't provide an upgrade path from bash. (This is also the biggest difference between Oil and fish, Oil and Elvish, etc.)

I talked with the authors of Ion on github almost a year ago. Ion was influenced by Oil, in particular this post:

http://www.oilshell.org/blog/2016/11/06.html

FWIW, POSIX emulation is on my TODO list: https://github.com/elves/elvish/issues/205

Not sure whether I will actually get to it though. Parsing sounds like a lot of headache. Your work is very impressive, but I’m not sure I will want to go through that... :)

Yeah, running ~/.bashrc is tough, and I'm not sure even Oil will get to it. Not only do you need parsing and execution of the bash language, but you also need builtins like bind and set -o vi (i.e. see "help bind").

Maybe bind is easier than I think; maybe it's just a few readline calls. I haven't looked into it.

But also, I noticed almost no shell scripts use extended globs like @(foo|bar), whereas the bash-completions package uses them EVERYWHERE. Completion scripts are almost like a different dialect than other shell scripts.

If you want to source /etc/bashrc on most distros, you're going to end up with thousands and thousands of lines of plugins from the bash-completions project.

The parsing is a big headache, but I think it forced me to learn how people use shell, which will benefit Oil.

It's been a full year since our alternative shells thread... might be time to revive it and see what's changed! Congrats on the Elvish releases!

Good point about bindings and completion scripts. Binding commands themselves are not big problems, the problem is replicating readline commands, there are loads of those.

I'm not sure how much work one needs to do to implement things like extended globbing, but my gut feeling is that once you can parse them it's more than half done. The functionalities are quite simple, as my impression goes.

I was mainly thinking about environment variables, aliases and wrapper functions when I said sourcing bashrc; that's probably what most people's bashrc files are mostly made of.

We totally should revive our alternative shells thread! I'll try to summarize my progress in the last year. Congrats on your progress with Oil too! ^_^

In the spaces you mention not hearing about fish are ones where you generally don't look for interactive applications/features. You also generally don't take a dependency like another shell when you are likely installing ruby or python already.

The two shells target different use cases. Hopefully they are both successful in attracting a good audience.

I've never met a devops person who wanted to write a 1,000+ line shell script. There are real languages that you can use instead.

Reply on news.ycombinator.com