Why Create a New Unix Shell?
oilshell.org
oilshell.org
1 year ago: https://news.ycombinator.com/item?id=22150603
Fish has its own quirks, some of which seem unnecessary, but it was a godsend. There is definitely still room for improvement of the old tools, a. lot. of. room.
(I'm the fish dev who implemented that)
Do you have any thoughts on why people don’t try it out? Where u also reluctant?
I write maybe a few dozen lines of code on a good day.
Fish manages to pull this off without _any_ configuration burden (I use it completely stock), which is exciting because usually to benefit from shortcuts or macros or other "productivity hacks" you have to become a power user.
> Every configuration option in a program is a place where the program is too stupid to figure out for itself what the user really wants, and should be considered a failure of both the program and the programmer who implemented it.
https://fishshell.com/docs/current/design.html#configurabili...
I often work with colleagues who ask how j get results so fast. They're very surprised when I say "remote emacs and a solid foundation with shell workflows".
When everything you work with is a text file, getting efficient with the shell just makes sense. Ask a carpenter about their tools and organization and you'll get a similar response.
There are plugins for zsh (and maybe other shells) which emulate this feature though.
If you already have Zsh configured well then Fish probably wouldn’t offer a huge advantage.
[1]: https://en.wikipedia.org/wiki/Fish_(Unix_shell)#Bash/fish_tr...
The downsides are similar to Mac. Less standard/compatible with other stuff. Zsh seems more like a traditional shell. Meaning it is far more complex. Fish really aims to keep simplicity.
I have used Fish for many years and swear by it but I am also not the kind of guy who writes long shell scripts. I use regular programming languages for that. For me fish is mainly about having a shell that works well in interactive use.
I recommend people Zsh: it's very familiar, scripts copied from the internet will most probably work and they have lots of flexibility to adjust it to their needs.
I personally have a Mac with Fish installed, I don't enjoy the process of sharpening my tools, I'm willing to give up flexibility in exchange of great defaults. It's the same reason why after I learned how to use a sharpening stone, I ended up buying an electric knife sharpener.
It has always baffled me the pushback that is given when someone works on projects like these. As if trying to do better isn't worth it.
https://github.com/zsh-users/zsh-autosuggestions
Now zsh feels just like fish, but with bash-compatible syntax (which was the main day-to-day problem I had with fish, otherwise it's fine, but except for the auto-suggestions I didn't really use any special fish features).
I sleep at night by writing stoic bash. If I encounter anything unexpected in a script I don’t try to handle it, I just report it and then crash.
Most of the bash contortions I’ve seen in my life happen when people (myself included) have tried to handle error conditions or ill specified inputs, gracefully.
A bash script is like a pre-flight checklist: a list of things that should work to get something going. Most bash scripts are like this: in general they are sequences of commands to change the system from one good state to another, not to recover it from a bad state.
In the analogy: if something is not ok with your aircraft it’s beyond the scope of the pre flight check to fix it. Don’t use the checklist as a way of trying to automatically detect what the error is in order to fix it.
I don't want my CLI to be a REPL for a scripting language. It's something I really disliked about PowerShell when I tried it. It just felt like a dotnet scripting language and not like a CLI.
If Oil provides a scripting interface with structured types etc. I'm all for it. I just don't want to have to deal with it when using the shell interactively. I love it when writing scripts and programs, but not for interactive shell use.
But powershell e.g. prints errors that look like stacktraces (it's been a few years i don't remember exactly) with type information. And while that may be desirable to some, when using bash i'm using a UI as a user and not programming and i don't want to read a stacktrace. I want an error message in english telling me what's wrong.
The lowest common denominator ends up being the good ol' bash (or sh)! Sure fish/oil is nice but quite a bit of mental gymnastics to keep them all straight in my head.
http://www.oilshell.org/why.html
And something I just released is a better "set -x", which no other shell has:
https://www.oilshell.org/release/0.8.7/doc/xtrace.html
So you could debug your programs with Oil if bash tracing isn't enough (which it often isn't for me).
Oil is also very easy to build and install -- all you need is a compiler, a shell, and make, and make will probably go away at some point.
for example, I always close browser windows and only keep a few tabs open at once, and then found out I was weird and many people keep huge amounts of tabs around.
My personal habit is to use bash, and always write shell scripts with #!/bin/sh as the first line.
Thing is - I've haven't tried other shells (since I was a student) because of "linux superstition". Same superstition that chooses filenames without spaces and prevents putting "." in your path.
Thing is - I think maybe decoupling the user interface from the scripting language should be more of a thing. People don't have as much trouble switching out their terminal emulator (a way of taking up the slack).
Maybe the way to a better UI while maintaining compatibility would be to choose a good "user interface" shell and maybe exporting SHELL=/bin/bash so scripts run?
Note that macOS still ships bash.
First off, any person who would write either a language or a shell, now or in the future, should read your page, and in fact, should read everything about your Oil shell/language...
It's a laudable effort!
This page is going to my HN favorites, for future review.
What follows next are my thoughts about selected excerpts of text on the page (please don't interpret as criticism, that's not the intent):
>"You can think of a Unix shell in two ways:
1. As a text-based user interface. You communicate with the operating system by typing commands.
2. As a language. It has variables, functions, and loops. Shell programs are text files that start with #!/bin/sh."
[...]
>"Are you reinventing Perl?
It's true that Perl is closer to shell than Python and Ruby are. For example, the perl -pie idiom can take the place of awk and sed. However, Perl isn't an acceptable shell either:
o It doesn't have true pipelines (of processes). I believe you have to "shell out" for this.
o It doesn't have a concept of file descriptor state. How do you write the equivalent of my_shell_func 2> err.txt, where my_shell_func can invoke both functions and external commands?
o Perl has a function called grep(), but the real grep is better for many problems."
PDS: This touches upon a historic problem, which is basically that either:
A) Shell designers work their way "up" to a programming language, in which case some elegant programming language features are either not thought about, or not thought deeply enough about, and thus either not implmented, or implemented poorly / "non-orthagonally", AKA a "Kludge".
Or:
B) Language designers work their way "down" to supporting shell commands / a shell subsystem -- in which case some elegant shell features are either not thought about, or not thought deeply enough about, and thus either not implmented, or implemented poorly / "non-orthagonally", AKA a "Kludge".
Also, languages are typically too heavy in what they require typed for shell-scripters, and shells are typically too light in what they allow (and error checking/debugging features) for programmers writing large programs...
If someone were to design the most ideal programming language which also can be used for shell scripting, they'd have to look at things from both perspectives, and they'd have to create a language/shell -- completely balanced "down the middle" between these two paradigms...
Perhaps the Oil shell/programming language -- is, or will become that middle line...
To that extent, I wish you a lot of luck!
If I were going to go down this path, I'd look at "first principles" (Elon Musk):
The greatest differentiator between shell commands and lines of code in most programming languages is the point you alluded to in your comparison with Perl:
"o It doesn't have true pipelines (of processes). I believe you have to "shell out" for this."
In other words, in most programming languages (unless you are using threads), you are guaranteed linear execution.
In a Unix shell -- because you can run multiple commands that may take varying amounts of time to complete, this guarantee is no longer present...
That would be the fundamental thing to keep in mind when writing this future language/shell...
(A good program to test when writing this language would be a Web-server -- where the base server is a single script, and then when it detects a connection, passes this over to either a single separate command-line invoked Unix program or script comprising multiple such Unix programs linked with pipes, but the central "server" script has to handle a whole lot of simultaneous I/O from multiple such separate programs (it centralizes database access!), and do this in a simple way that script programmers would be comfortable with, while guaranteeing the correct data flows...)
Anyway, wishing you luck in this endeavor!
To me, shell and Python/JS feel kind of similar, and that was sort of the thesis at the start of the project.
But if you just incrementally add features to Bourne shell, you basically get Korn shell, which is where bash got all the "bashisms" that people don't like.
This paper was surprising to me; the proprietary AT&T ksh was used for GUIs and so forth, AND it was embeddable like Lua:
http://www.oilshell.org/archive/ksh-usenix.pdf
[ksh] has the capabilities of perl and tcl, yet it is backward compatible with the Bourne shell. Applications of up to 25,000 lines have been written in ksh and are in production use.
Much of the impetus for ksh–93 was wksh, which allows graphical user interfaces to be written in ksh.
It's a lot more work to make something like Python or JS! Garbage collection is one issue; shells don't have it because they don't have recursive compound data structures or true functions.
So I would say that ksh "failed" because it took a lot of shortcuts. Perl, Python, Ruby, JS, and PHP won. So with Oil I also am upgrading shell, but it was a lot more work since I wanted to make it like the latter languages. It wasn't obvious at the outset how different these things are!
So in a sense Oil is upgrading shell to be more like the languages that won, which I still think is a good idea. It's weird that every computer boots to into a programming language REPL, but you're discouraged from learning it because it sucks. There's no reason that language shouldn't be a good one.
---
Yes pipelines are a big deal and the runtime is solid now so they can be enhanced. Ideas here, feel free to chime in: https://github.com/oilshell/oil/issues/843 :)
An excellent point!
>"Yes pipelines are a big deal and the runtime is solid now so they can be enhanced."
Maybe synchronous (aka, guaranteed linear) and asynchronous (no guarantees) are better ways to look at it...
For example, a simple shell command like 'echo' could be run either synchronously or asynchronously, but if we're assigning the result of it to some variable, it makes sense to run it synchronously, to get that result into the variable before proceeding to the next line of the program...
But, if we're running a disk defragmenting tool, or other process that we don't know when it's going to complete, then it makes sense to run it asynchronously.
Shell commands that are chained together via pipelines may run asynchronously, but the shell may wait for everything to complete to scoop up the output; so you've sort of have the entire line that was sent to the shell running synchronously (if you're assigning the result to a variable), even though parts of it are running asynchronously (with respect to one another)...
And of course, via ampersand, you could have the entire line run asynchronous to your program...
See, I'd almost say that one of the problems is that shell scripting languages almost have no knowledge (and correct me if I am wrong, I might be!) about the pipes they create inside of commands -- an ideal shell language would be able to know about these and hook these for various purposes, and specify what would be ran synchronously and asynchronously...
Or, another weird corner case -- what if you have a shell script that launches other shell scripts asynchronously (in subshells) and they launch other shell scripts also asynchronously -- in deeper level subshells?
How would your language/shell track/handle that case?
How do you communicate/track/determine when/where to assign results to things?
It's sort of like the language/shell -- must be able to :
A) Explicitly say what is asynchronous and what is synchronous...
B) For asynchronous things, be able to communicate with them, track their status, get results from them when they complete, etc., etc.
Now, of all of the programming languages in existence, I think Go is a good candidate for understanding synchronous vs. non-synchronous -- but you don't exactly get an "easily scriptable" beginner-friendly shell-scripting language with Go...
Python seems to be the right balance between "ease of use" and "gives you control".
So I'm agreed with all that you've said about Python, and your use of Python in code...
Also, dgsh is along these lines: https://www.spinellis.gr/sw/dgsh/#compress-compare
It appears to use Unix domain sockets, not pipes.
I think I might want to use the Ruby-like blocks for syntax, something like
grep FOO *.py | wc -l | sort -n
pipeline {
pipe :p1 # variables that are pipes
pipe :p2
# this syntax isn't great but shows the idea
grep FOO *.py > &p1
< &p1 wc -l > &p2
< &p2 sort -n
}
Although I think richer pipelines make sense, structured data will probably come first:
All Unix (and derivative OS utilities) typically pass unstructured raw data to one another via pipes...
That's sort of a good thing and a bad thing at the same time...
It's great for such things as binary files, binary streams, anything binary in nature...
But equal-and-oppositely, it's terrible for say, if I wanted to put the output of a 'ls -al', and I wanted to put all of those fields into something like a structured XML or JSON or BSON or what-have-you.
I also like your point about FD's for pipes... that's a good idea too.
Yes, I could see deadlocks as possible, but then, maybe yet another extra step of engineering/forethought is necessary -- how do we manage locking in those scenarios? Maybe we could create timers that automatically break a deadlock after a certain number of seconds, and/or put the lock explicitly under program control (like the FD's for pipes), etc., etc.
Anyway, some fascinating stuff!
>"Although I think richer pipelines make sense, structured data will probably come first:
https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil"
Looks like you've been doing your homework! <g>
Some notes here about interactive shell ideas: https://github.com/oilshell/oil/wiki/Interactive-Shell
Some people might think Oil is sort of a text-only or retro project ... but it really isn't, it's a SPEED project. I use shell because it's the fastest to get certain things done.
But other things are faster when done with GUIs. Actually I wrote the last 5 or 6 blog posts in https://stackedit.io because copying and pasting huge swaths of text and links and images is faster with the mouse!
I am a die-hard Vim person for code, but I realized that "thinking and writing" is aided by the mouse and by rich GUIs (but not slow ones, which is hard!). Previewing hyperlinks is also very important.
People also ask the same question about Windows, and even DOS before it; there you actually could do it from a batch file, since CMD.EXE/COMMAND.COM effectively sources all batch files, it doesn't run them in a subprocess – but the same issue occurs if you write a program in some other language than batch, it runs in a subprocess and so can't modify CMD.EXE/COMMAND.COM's environment. (People resorted to some tricks though, like having a BATCH1.BAT call their program, and then afterwards call BATCH2.BAT, and their program modifies BATCH2.BAT on disk).
Actually on DOS, there is a way to modify COMMAND.COM's environment – COMMAND.COM installs an undocumented interrupt, INT 0x2E, which you can use to send commands to COMMAND.COM to run. So you can actually modify COMMAND.COM's environment. (Only the root COMAMND.COM installs a handler for INT 0x2E, nested ones do not.) Of course, due to no memory protection, there are also nasty ways of doing this, like modifying COMMAND.COM's memory. (And changing COMMAND.COM directory isn't an issue, since under DOS, the current directory is system-wide, not per-process.)
I was thinking, you could do something like "INT 0x2E" in a Unix shell. Create a Unix domain socket, and the shell listens for commands on it to execute. Put the path to the socket in an environment variable which is inherited by subprocesses, e.g. SHELL_CONTROL. Then, a subprocess can inspect and modify the shell's environment, working directory, functions, aliases, running jobs, etc, by reading/writing the Unix domain socket mentioned in SHELL_CONTROL.
https://github.com/oilshell/oil/issues/738
I would call the 2 possible solutions an API or IPC, and what you're describing is basically IPC.
The devil is in the details though... I'm interested in a prototype :) A consumer of the interface will help shape it.