A Tour of the Oil Language | Hacker News Reader

A Tour of the Oil Language

oilshell.org

159 pointsdmart42 comments

I'm interested in and excited about oil, but three things have stopped me every time I've tried to give it a go.

1. The complete and utter lack of useful documentation. There's not even a link to any docs from the main page of the shell's site. You have to get to the docs from the release page, [1] but I had to find this out by Googling. Given the number of blog-post-like entries in the docs, seems like it would have been less effort and more effective just to write standardized and complete documentation from the start.

Just as an example: can anyone tell me what built-ins Oil has, how to use them, and where they are documented? I'll give you a hint. I found out from one of the blog posts that oil has "min" and "max" commands. The following is valid Oil code:

    const x = min(1, 2)

But what other commands does Oil have, and how do I use them? I have no idea. Some of the doc pages 404 so maybe they're supposed to be there. I have found some pages listing built-ins which are mostly undocumented, but none of them list min or max, so I know they're not complete.

2. It's still pretty buggy and lots of features are missing. Development has seemingly moved slowly over the last several years. If it ever did get to the point where I could make it a daily driver then I'd be willing to track down and report bugs, of course.

3. I'm a little concerned by how much emphasis is put on POSIX / Bash compatibility. Unless there's going to be a way to fully disable what Oil considers "legacy" language support in a script, then that is going to leave a ton of footguns for people who want to write "modern" Oil code. It's not clear to me whether that's the plan or not (I haven't read every one of the blog posts), but it seems like a mistake to me to sacrifice possible syntax improvements on the altar of compatibility.

To be clear, I realize this is basically someone's solo project, and I am impressed by how far it's been taken, just clarifying why I don't think it's ready for general use yet.

[1] https://www.oilshell.org/release/latest/doc/

Is its goal to be "a shell language that's decent to program in"?

Yes, the home page says that it's for Python and JavaScript users that avoid shell: https://www.oilshell.org/

Also see The Simplest Explanation of Oil https://www.oilshell.org/blog/2020/01/simplest-explanation.h...

In addition to being "decent" / familiar / learnable, it's also more powerful, as bash is showing its age and lacking features many users would like (recursive data structures, JSON, etc.)

(author here) I don't think anything you wrote should be a surprise to regular readers of the blog. Many of my blog posts in the last year say "the scope of the project is too large" and similar stuff.

http://www.oilshell.org/why.html#why-not-use-it

It might be a surprise if you're only reading what shows up on news aggregators. But to be clear, I basically write about what I'm thinking, what progress has been made, etc. And I do "documentation-driven development", which is IMO the best way to design a consistent and coherent language.

The main purpose is to get people to test out the shell, and attract developers who want to push in the same direction. I like this way of working, as I've gotten a lot of valuable feedback and contributions on both OSH and Oil. And I can use more.

Oil is NOT a "product" which you can download and use. (It doesn't even have a logo, partly for that reason.) Right now it's better for say people who have the desire and ability to make it through the quick start for contributing:

https://github.com/oilshell/oil/wiki/Contributing (how to get a working bin/osh from git in 30 seconds on most Linux distros. You can then hack on it in pure Python and it will be fast)

----

The "summer blog backlog" has some project status updates that aren't out yet, but I'll give the tl;dr version here.

There are roughly 5 equal sized parts of project, all large:

    1. The compatible OSH language (mature, NOT buggy, but not fast yet)
    2. The Oil language (this is 2 years old; there are bugs due to the way we reuse CPython)
    3. The interactive shell (punting this to "headless shell")
    4. Semi-automatic translation of the "executable spec" from Python to C++
    5. Documentation

For context, last October I wrote a post saying that four features (that are DONE and polished) justify the OSH language and I can punt on the Oil language until the indefinite future.

https://www.oilshell.org/blog/2020/10/osh-features.html

So that was one way of cutting scope. But basically I changed my mind based on prior successes. So I worked on the Oil language a lot this year, and I'm pleased with how its coming together. (The summer blog backlog has sketches of new features. Big things left: the "missing declarative part" of shell for configuration/not-YAML; functions/coprocesses as opposed to shell-like procs)

I made a big list of alternative shells, and none of them is what I want, or what I think the open source ecosystem needs:

https://github.com/oilshell/oil/wiki/Alternative-Shells

This is a very long argument and has to do with the "Perlis-Thompson Principle" and "narrow waists", which I'm about 25% through explaining on the blog.

-----

So the other way of cutting scope is punting the interactive shell into another process, with Oil being a "headless shell":

See these two comments for details:

https://lobste.rs/s/xestey/opinionated_guide_xargs#c_cn4r2i

https://lobste.rs/s/sdum3p/if_you_could_rewrite_anything_fro... (yes I need to make blog posts out of these)

To clearly repeat what's in those posts: you won't get more than a basic interactive shell based on Oil unless someone else makes big effort. Join #shell-gui on Oil's Zulip, etc.

----

So the project is now focused on #1 #2 and #5: OSH and Oil and documentation. The translation to C++ passes about half the tests, but it has stalled. So I have to come up with some way of getting that done too, and it might involve soliciting donations to hire someone, some kind of two-sided market etc.

But right now I'm just concentrating on the Oil language design and explaining it. So you might see many more compelling posts about things you won't be able to immediately use. That is on purpose!

It all absolutely works -- download it now and try it -- but there is a gap to bridge to make it production quality, which involves documentation and translation to C++.

(This post is from a year ago but may give some color on these issues: https://www.oilshell.org/blog/2020/08/risks.html)

In addition to direct collaboration, I also encourage parallel efforts. Someone started porting Oil to Nim, and another person wanted to rewrite Oil only (not OSH) in Rust, etc. I encourage all those efforts.

The project is long, but the "hard part" OSH is quite mature (you should test it), and the "exciting part" Oil is less than 2 years old. And five years ago people said it was impossible to evolve bash into a good language. Now people seem to be taking it for granted that it should exist :)

Thank you for the extensive and thoughtful comment! This does help clarify your approach quite considerably. I wonder, since you are hoping to attract collaborators, whether there is some kind of formal spec for the language somewhere? For example, you mentioned parallel efforts: suppose I wanted to write a port to pure C; is there any way, short of reading every one of your posts and trying to contain the whole language in my head at once, for me to know exactly what I need to implement?

Something I've been trying to figure out: what is the exact relationship at present between OSH and Oil? When you say "OSH" do you mean the language, or the shell itself "oil shell"? If Oil is not something I can download, why exactly does that `const v = max(1, 2)` statement work in osh? It's clearly not just a Bash implementation, it's got other features. Is that a subset of Oil's features? Which subset?

Since you're also interested in other shells, you might have a look at pyp [1]. It captures a lot of the way I personally would like to use some future shell. If the features of pyp were integrated into the shell itself, you wouldn't need an external command, you could just (for example) pipe the output of one program into a python-like statement that mangles the incoming strings in some way, and pipe that out to some xargs-like program to use in a subshell. (The fact that you apparently can't use the pipe in what Xonsh calls "Python mode" is for me the central limiting feature of that shell.)

[1] https://github.com/hauntsaninja/pyp

I meant to write a blog post about implementing the Oil language, but this blurb gives a lot of info!

https://www.oilshell.org/blog/2020/07/blog-roadmap.html#how-...

The algebraic data types alone give a huge leg up, but there's more than that. The spec is the code! That's why it's written in an unusual style with a high level language:

http://www.oilshell.org/blog/2019/06/17.html#why-is-it-writt...

At the end of every release there are some links to key source files. At some point these could be nice HTML diagrams rather than raw source code, but the info is concisely encoded:

https://www.oilshell.org/release/0.9.2/ (scroll down to "Source Code")

Also, I don't expect much of this to be intelligible at first... It's a bunch of pointers to start. Anyone who wants to go down this path this will have to ask a lot of questions on Zulip, which will motivate updates to the docs.

-----

I will write a FAQ about OSH vs Oil, since that has come up several times, and it has evolved over time.

There is only a single binary and single interpreter, but there are two entry points bin/osh and bin/oil, which basically correspond to the OSH and Oil languages.

So the interpreter is parameterized by parsing and runtime options. They are all under this group:

    shopt --set oil:all  # turn on everything as if you were running bin/oil

That is what this Tour doc refers to -- a shared interpreter data model, and a shared process model.

So we talk about them and document them as separate languages. But really it's a graceful upgrade of a single language. If you want you can even set individual options like shopt --set parse_paren.

-----

The syntax

    const v = max(1, 2)

is valid in both bin/osh and bin/oil, since nobody has complained the taking 'const' breaks anything. (We try to minimize new keywords -- e.g. 'pass' was removed from the language based on feedback). Whether you want to consider it OSH or Oil is up for debate :) The meanings of OSH and Oil have slightly changed as the project evolves, but OSH means roughly "compatible features" and Oil means "new, incompatible features". It's a spectrum.

(I think I have written all of this in various places, but it does need to be put in a more central place on the site)

----

pyp looks interesting, I added it here under Awk-like DSLs: https://github.com/oilshell/oil/wiki/Internal-DSLs-for-Shell

The linked Tour is part of the documentation, which at a glance looks extremely extensive. You mention some issues; are you saying these docs are too hard to discover and lacking a good-enough TOC or index ?

In addition to them being both of those things, they are also, so far as I can tell (which is incomplete because of the lack of an index), missing documentation for most functions. If you click through to some of the other documentation for parts of Oil, you'll see that they're mostly written in a "blog post" style, with the intent of giving a reader a brief tour of a couple of features, while leaving everything else completely undocumented.

For an example of good documentation, look at the docs for the Python OS module. [1] You'll see that every single function in the OS module is listed, on order, with clarifications on its usage, what values it returns, what arguments it needs, and gives clarifying examples where needed.

For a project that's much younger than Python (though by no means "new"), you don't expect quite as much. But to compare: the page documenting the command language 404s [2], the help file that I believe ships with osh documents a few commands but has huge missing gaps (and doesn't mention min and max at all) [3], and the page on strings has more of a theoretical discussion on strings than practical documentation and has major sections labeled "todo" [4].

[1] https://docs.python.org/3/library/os.html

[2] https://www.oilshell.org/release/latest/doc/command-language...

[3] https://www.oilshell.org/release/latest/doc/osh-help.html

[4] https://www.oilshell.org/release/latest/doc/strings.html

Yeah the reference is supposed to be "OSH Help Topics" and "Oil Help Topics", which are linked from this page:

https://www.oilshell.org/release/0.9.1/doc/

They list everything concisely on one page. But the body of the docs is mostly not done.

https://www.oilshell.org/release/0.9.1/doc/osh-help-topics.h...

https://www.oilshell.org/release/0.9.1/doc/oil-help-topics.h...

The release/$VERSION/ page lists the docs that are done, while /release/$VERSION/doc/ has all docs, including ones in progress.

Anything that's in progress has a pink banner at the top.

I got help filling out all the OSH builtins last year, and can use more help.

----

The 404 is an error which I noted here 2 days ago, and will be fixed in the next release: https://oilshell.zulipchat.com/#narrow/stream/121540-oil-dis...

If you see any errors like that in 0.9.2 or later, please file a bug. Also send feedback about things you would like documented. (But some things simply aren't done enough to be documented, like the list of builtin functions.)

Super duper excited about this and have been for the past five years. Awesome to see things coming together.. good things take time! This guy is singlehandledly trying to elevate shell, revealing its secret beauty and immense power. My favorite thing is his obsession with bash backwards compatibility-- it has led him to get super deep into the bash parser. Unrelated but useful by the same person: Bernstein chaining of ssh and su: https://www.oilshell.org/blog/2017/01/31.html

Note that the Oil shell is also bash-compatible [1]. This project (new shell intepreter and shell language) is not lacking in ambition, that's for sure. Not a heavy shell user myself but will be trying oilshell on some sandbox instances in the near future.

[1] https://www.oilshell.org/cross-ref.html?tag=osh-language#osh...

I would be especially interested in a comparison with Zsh, which is also very much a "better Bash".

I know you're looking for a comparison of the "new" syntax of zsh and oil but I think it's worth noting the biggest difference is probably going to be Zsh is an extended sh shell with a bash (well ksh) emulation mode capability whereas oil is a shell where the language is a superset of bash syntax.

Zsh tries to be similar to bash but it has no problem not aiming to be 100% bash in normal mode. Osh wants you to be able to replace #!/bin/bash with #!/bin/osh and run your script without setting the mode to something special. Then when you're ready to extend it you can write either bash or osh without toggling emulation mode or worrying about which syntax or behavior differences you'll run into not being in emulation mode.

The progress of Oil Shell is looking really good and hopefully it can reach the stable 1.0 version soon.

I've been following Oil Shell for several years now by reading the blog articles. The fact that it has managed to adopt and adapt some of the best features of other popular (mostly) dynamic programming languages into a cohesive scripting language and at the same time being backward compatible with Bash is amazing.

Recently there was an HN post and discussions on Linux Router unified command written entirely in Bash [1]. When reading the discussions, the first thing that come to my mind was that it will great if this can be somehow re-written in Oil Shell. But why stop there? What I'd like eventually is a seamless wrapper that maps the native related Linux tools in Oil Shell that can provide similar Cisco IOS like commands, configs and interfaces. It can then fashion the Linux kernel into a glorious network OS without writing extensive separate native networking system services as Quagga, LiSA or Cisco NX-OS. Heck the entire NX-OS can probably be cloned with Oil Shell on top of vanilla Linux kernel and eBPF. Before someone pointed out that what so special about Cisco IOS, it is by far the most popular network OS conventions to the networking professionals, and most of them are very familiar with the configs and commands. Due to the increasingly popularity of containers and Kubernetes, robust Linux networking configuration and automation tools will be at the center of this latest trend. Always remember that Linux once started life as a poor's man clone to the very popular UNIX System V itself.

[1]https://news.ycombinator.com/item?id=28512768

Hm interesting, yup that is exactly the kind of program that might get "too big" for bash, and running it under Oil is a good idea. You can then incrementally upgrade to Oil -- no big bang rewrite required!

Other huge shell programs here: https://github.com/oilshell/oil/wiki/The-Biggest-Shell-Progr...

And yes Oil should be suitable for writing command line tools (although it needs a modern flag parser, not the ancient getopt builtin that shell has.) Often the right thing to do is to write a tiny bit of C to interface with the kernel, and then call that from shell. That is very useful for Linus container syscalls, and I can also see it being useful for low level networking. I would love to see a project in that direction :)

I'm writing some posts now about project plans! Thanks for the support.

I see Oil and oilshell posted here somewhat regularly.

Is anyone using it and can relay their experiences?

Is it ready to be used as a daily driver?

(author here) I'd say it's better now for people who feel comfortable contributing (or just want to test it out and send feedback). Some parts of the project are mature (OSH language) but others are less mature (Oil language).

Long reply in this thread that gives details: https://news.ycombinator.com/item?id=28547185

> Some parts of the project are mature (OSH language) but others are less mature (Oil language).

In a way you've placed the opposite of the bet of most of your 'colleagues' in the field of pioneering new shells. To me, the other way around is more tempting because novelty is fun for you, the developer, and innovation here is the main attraction. Not to mention that lots of people might've burned out in the 'Osh phase', as it were.

I hope that taking the long way around pays off for you, because I know it's been arduous! One advantage it might give Oil, given the attention you seem to pay to other projects in the same space, is that with the Oil language you can now also draw from other next-gen shell languages which have had opportunities to grow small communities and undergo revision in light of their use in the tome job were focusing on Osh.

I really like some of those other projects, so of course I hope there's a place for them going forward as well (maybe as my own login shell, even!). But I'd love to see Osh earn an installed-by-default place on some major operating systems/distributions some day, and in that way open the door to widely distributed shell scripts in a more modern shell language in Oil.

I look forward to really digging in to this new doc and writing some toy scripts. :)

Yes Oil is the only shell that's a graceful upgrade from POSIX sh and bash :) If it were done the other way around, that would have been impossible.

As long as the implementation makes enough progress, I have no doubt it will pay off. I think this is "obvious" if you look at the history of technology, but I guess I have to publish this blog post draft called Don't Break X (Why Oil is a Long-Winded Project)

- Don't Break Windows apps. https://www.joelonsoftware.com/2004/06/13/how-microsoft-lost...

- Don't break JavaScript and the web. From HOPL IV. https://hopl4.sigplan.org/details/hopl-4-papers/10/JavaScrip... . Wirfs-Brock and Brendan Eich discuss whether the JS standards committee did the WRONG THING for 10 years.

- Don't break the Linux kernel interface. https://unix.stackexchange.com/questions/235335/why-is-there... (there is probably a better link for this one; there's one with a lot of profanity and insults from Linus Torvalds that I'm not sure I want to use)

Similarly, Oil doesn't break shell, and we definitely need a shell like that.

Other alternative, incompatible shells are valuable, but there's absolutely no question that there needs to be a compatible upgrade. Just like C++ has made huge leaps while Rust was being developed. (Crucially, Oil is different than C++ as it actually deprecates some legacy with parsing and runtime options.)

------

There's a distortion where all the people interested in Oil now are by definition interested in new stuff. That is, in reality 95% of people are more interested in OSH than Oil, because they don't care about new languages and they just want things to work. But the people I hear from are the 5% who care about new stuff. (which is good, but it's not representative).

A similar distortion is that there are probably 10x to 50x more people working on C++ than Rust (e.g. at least 10,000 people at Google alone; I'd surprised if more than 100 or 1000 do any Rust for work. Also consider older companies like Oracle.). But if you were to go by this corner of the Internet, you might think Rust is the more popular language.

In reality C++ is the far bigger and more capable ecosystem, if you care about robotics, embedded, AI, GPU, native desktop apps, information retrieval and data structures, etc.

What are the main reasons, if any, for users to prefer Osh to Bash today, even if they don't intend to use or learn Oil for the foreseeable future?

This page needs an update but covers that to an extent:

https://www.oilshell.org/why.html

It's mostly useful as a dev tool right now. I would also note that probably 90% of bash users don't do any bash programming. As mentioned elsewhere in the thread, Oil is very much focused on the programming part.

It has a better foundation for an interactive shell than bash, but somebody else needs to push on that part via the "headless shell".

I think it comes up often because:

- A lot of people here dream of better shell languages

- Oil is very ambitious in that regard

- Oil has a lot of in depth and well written articles about itself, and shell languages in general

- Oil creator is active on HN

   var person = 'alice'
   echo "hi $person, $(echo bye)"  # => hi bob, bye

This is a strange language that turns alice strings into bob strings (quoted from article)

Is it possible that this is a typo in the example?

Thanks to Paul for fixing this just now :) https://github.com/oilshell/oil/pull/988

For google-ability did you really need to occupy Oil Shell when Shell Oil is very much already a thing?

Seems like search engines are smart enough to distinguish them.

https://duckduckgo.com/?q=oil+shell

I can't tell if you're joking, all of the results except the top one show "Shell oil" results for me. Not hard to imagine that this impacts searchability if you're looking for something specific.

Heck, sometimes I search for oil shell when I’m looking for shell oil. I wouldn’t even want a search engine to know about order.

Yes, but the first result for this one is oil shell, whereas if I search the other way around it isn’t even on page 1.

I wonder if people will end up googling it as oillang (or even oilang), similar to what happened with golang.

Wouldn’t it be oillang?

Let's hope so, because 'oil language' is also going to be problematic: https://www.google.com/search?q=oil+language https://en.wikipedia.org/wiki/Langues_d%27o%C3%AFl . I came here half-expecting an article about medieval French.

A metric by which the optimal language is AmbidextrousScallywag

Will oilshell have job control? I couldn’t quite tell. A lot of small shell projects seem to neglect it.

It has basic job control, e.g. Ctrl-Z and fg work. Full job control will probably have to wait for a contributor (I use tmux instead of job control, and job control is very hairy). If the rest of Oil becomes popular I'm sure that will happen.

Oups, title wasn't an English translation of langues d'oïl!

The tech aside, this is horrendous naming! The oil language puts in my mind the image of "oily greasy hair", commonly associated with someone sly and contriving.

I take it you are not a fan of Scheme and Racket?

Haha, and python too? But no, those all I don't have a problem with. With Oil, maybe Harry Potter (Snape) has something to do with it.

As HN is covering so many different subjects, I thought that it was related to these oil languages :) https://en.wikipedia.org/wiki/Langues_d%27o%C3%AFl

This might help: http://www.oilshell.org/blog/2019/06/17.html#why-is-the-proj...

Reply on news.ycombinator.com