Elvish is a friendly interactive shell and an expressive programming language | Hacker News Reader

Elvish is a friendly interactive shell and an expressive programming language

elv.sh

149 pointsfmakunbound49 comments

Elvish author here, AMA :)

Hello. Looks amazing. How does the command completion compare to fish? That's the feature I like about fish because it seems very predictive. But just the other day I needed to parse JSON from curl or whatever and ended up writing a Node.js script which would have been unnecessary with Elvish.

Elvish has a really simple API for writing completers (https://elv.sh/ref/edit.html#completion-api). The simplicity of the API stems from some fundamental language features like first-class functions, value pipes and composite data types. In more traditional shells without these language features (including fish), the completer API is usually made up of obscure naming conventions, specialized command for outputting candidates and controlling the format, etc.

I think Elvish's simple completer API (which is very similar to completion protocols used by editors, FWIW) is proof that serious programming capabilities is actually quite essential for a shell. You wouldn't write editor plugins using bash, but people have to write completers and other shell plugins in bash all the time.

However, what would affect your day to day experience the most is probably the coverage of existing completers. There are some community efforts you can find in https://github.com/elves/awesome-elvish#completion-scripts, but it's quite clear that Elvish is lagging behind.

This is mostly because I'm still trying to figure out a lot of things in the language itself, and I don't want people to invest a lot of completers before they get broken by language changes. Once the language becomes more stable (which will perhaps take another year or so) I'll invest much more in this area.

> "There are some community efforts you can find in https://github.com/elves/awesome-elvish#completion-scripts, but it's quite clear that Elvish is lagging behind."

clap-rs, which is a popular command line parser for Rust, can generate Elvish completion scripts automatically and it works really well. Maybe you want to add that to the list.

The API looks awesome. But when I tried it just now, I see no color in the terminal, and it does not have anything like the completion I see in fish.

I don't know if you have tried it but fish basically predicts what you are going to enter based on the directory you are in and what you have entered before and gives a preview as you are typing, and then you can hit the right arrow to complete it.

Since that is extremely convenient for me and your shell seems to not have color (at least not in my Ubuntu terminal right now), I am going to stick with fish.

This is really exciting. I've been using zsh for years but would probably change to this shell with better completions and theming.

How does Elvish compare to Powershell, at least in philosophy?

Elvish and PowerShell shares one big thing: let command outputs carry structured value. Everything else is quite different:

* Elvish is a more minimalist language influenced by the Lisp tradition, whereas PowerShell is a much larger language with its own distinct "flavor" (in a good way; the language has some quirks but seems decent overall).

* PowerShell has a lot of integration with the .NET ecosystem and Windows APIs. Elvish obviously has none of those.

* Elvish has a pretty sophisticated UI, whereas PowerShell seems to lag behind a bit in UI features.

Getting back to philosophy, it seems to me that the development PowerShell is largely driven by facilitating common Windows sysadmin tasks, and the team seems to have some very concrete use cases in mind.

On the other hand, the design of Elvish has two main tenets, one theoretical and one practical: make a good programming language that is still "shell like", and make a powerful flexible UI suitable for daily use.

> Elvish is a more minimalist language influenced by the Lisp tradition, whereas PowerShell is a much larger language with its own distinct "flavor" (in a good way; the language has some quirks but seems decent overall).

Could you expand on what you mean by this? It sounds like you're conflating the size of the language with some measure of lispiness, but Common Lisp for example is huge.

It's confusing to me that you don't consider LISP to be a simple/minimalist language.

Core language, not standard library, extensions, etc.

Simplicity of core language and syntax does not necessarily translate into simplicity of the language, see Common Lisp.

So, trying out elvish a bit tonight, I've noticed that there are a lot of little quirky places that whitespace is significant in elvish? One example is lambda syntax. Do you have any plans on making that less of a thing, or is it here to stay?

I dabbled a bit with Elvish recently and enjoyed it. What I found harder than I expected it to be was getting an initial configuration to my liking. While the documentation is excellent I find that it lacks some kind of beginners tutorial for plain users. The Fundamentals section is a good intro to the language but what I wanted to know more as a newbie was that my startup file is expected to be ~/.elvish/rc.elv, for example. It's all there but the basic info could be more prominent in the docs in my opinion. Otherwise I'm happy with Elvish, thanks for the good work.

Is there a road map?

kudos for the website

Thanks! Is there anything you particularly like, and anything that can be improved?

Amazing! Built-in file manager and folder history are wonderful, but it doesn't set the pane title in tmux correctly out-of-the-box.

> it doesn't set the pane title in tmux correctly out-of-the-box

Yeah, it's an unfortunate side effect of Elvish not having job control at the moment; tmux (and terminal emulators) use the information about the foreground process group to automatically set the title, but since Elvish doesn't have job control and stays in the foreground, that doesn't work.

However, there is a module that does this: https://github.com/zzamboni/elvish-modules/blob/master/termi...

FYI for Windows users, no Unicode

https://github.com/elves/elvish/issues/1124

Or you could get fzf innyour current shell for most of the features (the command history search, file preview/open, etc). Familiar syntax is a big plus though, a point on which most shells leave a lot of desired.

If you want familiar syntax, Oil is the most bash-compatible shell by a mile (which means it's also POSIX compliant).

But you can opt in to a bunch of fixes for longstanding shell warts, shown here:

https://www.oilshell.org/release/0.8.0/doc/idioms.html

So upgrading is / will be worth it.

-----

There's an issue to run fzf, which was partially done last year: https://github.com/oilshell/oil/issues/322

I could use help looking into what remains there!

And there are more such issues here: https://github.com/oilshell/oil/labels/should-run-this

Just running these scripts and localizing the error is a big help! And that would get us closer to a new shell that can be realistically adopted.

Oil has run thousands of lines of unmodified bash scripts, and many thousands more with minor patches, including some of the biggest shell scripts in the world:

https://www.oilshell.org/blog/2020/04/release-0.8.pre4.html

> Oil is the most bash-compatible shell by a mile (which means it's also POSIX compliant).

Bash is not POSIX-compliant by default; it has a POSIX mode (https://www.gnu.org/software/bash/manual/html_node/Bash-POSI...), which changes a lot of behaviors. It's mostly small things, but it's a long list nonetheless.

I wonder what's Oil's approach here - does Oil has separate "bash mode" and "POSIX mode"? Or are the divergences inconsequential enough for most scripts that you don't bother distinguishing them?

FWIW, I avoid using the word "POSIX-compliant", because strict POSIX compliance is usually not exactly what people want when they use this word. Instead I'd just say "POSIX-ish" as a vague notation for the intersection set between bash and the actual POSIX standard, and maybe more minimal POSIX-ish shells like dash.

Yeah I agree it's not a great phrase, but people seem to have it in their heads.

What I mean is that although Oil is very bash-compatible, and that's how I usually describe it, it's not ONLY bash compatible.

Oil's spec tests run against multiple shells for this reason. I try not to reproduce the bugs of bash unless they are necessary to actually run something. (And this happens a lot -- there are accidental corners that nobody uses.)

For example, the Linux distro milestone in 2018 showed Oil's portability: Debian's debootstrap runs with dash, Alpine's abuild runs with busybox ash, and Aboriginal runs with bash.

But Oil runs all 3 of those scripts by default, unmodified. For example, the Alpine one used alias, which is off by default in bash, but on in POSIX mode. Oil has aliases on for this reason (which makes it more POSIX compliant than bash in that regard.)

----

I also test with the Smoosh POSIX test suite, from the Smoosh formal semantics (though that is not the only suite the paper uses to judge compliance.) On that suite, Oil passes more POSIX tests than all shells except bash now. And the differences are mostly unimplemented features, so I think it's inevitable that it will pass more than bash.

So I still believe in the assertion here: Oil is actually a better POSIX. It describes what shell scripts require in practice, better than POSIX does.

http://www.oilshell.org/blog/2018/01/28.html#limit-to-posix

----

So Oil doesn't have a POSIX mode yet. It might in the future. So far it doesn't seem necessary to run real shell scripts.

But yes I agree that "POSIX by the letter" isn't a super useful property. What I mean is that Oil is designed to run a wide variety of shell scripts, and also help people write portable code.

I think when people say "POSIX" they mean "I want my program to run under multiple shells", and Oil definitely supports that.

Thanks for the Oil shell suggestion. I will definitely check it out this evening.

I don't care for POSIX compatibility. I'm a long time Bash user, I've written programs I probably should not have in Bash, and I'm familiar with most of the pitfalls described in a recent HN post https://news.ycombinator.com/item?id=24401085 however, after reading through them I thought this is some serious bullshit one must contend with and went looking or alternate shells. Elvish being one of them.

If you've written programs you shouldn't have in bash, then you're very much the target user for Oil :) As stated on the home page, it's our upgrade path from bash to a better language and runtime.

So I would be interested if those programs run under Oil. Many do, and if they don't, the fix is usually small. Other programs I've run include Lisp and brainfuck interpreters in bash. And thousands of lines of bash completion scripts.

If the programs run, you can opt into strictness checks to improve them / find bugs. And if you want to drop bash compatibility, you can use the new Oil language. (still in progress, but you can try it right now)

The goal of Oil is very much to REMOVE all the pitfalls in that thread! It's done with option groups shopt -s oil:basic or shopt -s oil:all. Feel free to contact me on Github or Zulip about it. If you have bash programs, that makes it more interesting. I'm looking for feedback to help shape the Oil language :)

Based on my experience I think only 10% of bash users write a program that's longer than 5 lines, or use any bash features. Most people just use what their distro provides (which is what I did for years as well).

A link to some related fun & craziness, for people not too familiar with Oil:

https://github.com/oilshell/oil/wiki/The-Biggest-Shell-Progr...

The buried lede here is that Oil uses some kind of tool to translate Python to C++ before running it. This sounds super interesting. Also am I misreading or is Oil written in Python 2.7?

There are 2 builds now: the one that runs under a slice of the Python 2 interpreter, and "oil-native", which is that code translated to C++.

Rather than saying Oil is written in Python 2, it's better to say that Oil is written in ASDL + a subset of statically typed Python + an abstract dialect of regexes (similar to Eggex).

In other words, it's a bunch of custom DSLs. And all of those DSLs translate to Python 2, as well as C++. So yes there is a lot of metaprogramming going on!

This makes the code shorter, more correct, and even pretty fast (slightly faster than bash, and it appears we will maintain that, although it may get slower in the short term)

Resources here, happy to answer any more questions:

http://www.oilshell.org/blog/2019/06/17.html#why-is-it-writt...

http://www.oilshell.org/blog/2020/03/recap.html#mycpp-the-go...

FZF is amazing and I highly recommend it as well, but this project seems to go beyond what FZF provides.

This bit in particular:

    curl https://api.github.com/repos/elves/elvish/issues |
     from-json | all (one) |
     each [issue]{ echo $issue[number]: $issue[title] } |
     head -n 11

That being said I'm still not willing to switch to a non-standard shell syntax, too difficult to use third party servers after that.

Also un*x shells are inherently limited by the underlying concept of pipes exchanging only byte streams with no metadata or higher level constructs. You can hack your way around that (as this project seems to be doing) but it's always going to be kludgy. In particular you can't expect any third-party command to play by your rules.

Eh, last time I messed with I lost interest after noting that from-json | to-json would silently convert all numbers into quoted strings and do terrible things when nil was involved. It made non-trivial JSON manipulation not worth it. They might have fixed it by now.

Yes, Elvish now has a dedicated number type, so conversation from JSON is now lossless.

Excellent -- nil handling work as well?

Yes, Elvish's $nil maps to JSON's null.

presumably you can get jq to do something similar, well enough for 95% of the time, which is the issue. like you, i'm not saying new shells don't have a place, but adoption is really the problem.

you can get around this if the features are compelling enough. personally, i like fish. certainly, it was a rough adoption curve, but worth it IMO so i can stop setting up and tweaking zsh, plus the built-in auto-complete is fzf-ish enough for me. and i'm told people use powershell even on non-Windows systems. it's just a very tough sell.

> too difficult to use third party servers after that

Just "scp" the binary to the server and you're good to go :)

Yeah this one of the selling points, IMO. It's a gigantic Go static executable and easily available for a wide variety of architectures.

fzf is indeed a very neat standalone tool, and for the use cases it has been optimized for, it is superior to Elvish's UI in many aspects. However, Elvish also has a very expressive programming language, and eventually it will offer more options in programmability.

This looks really cool and useful. Does anybody know how this compares to NuShell?

This looks really cool. I'd be curious to see a video demo of the file manager, directory, and command history UIs.

is it posix script compliant?

No. If you are looking for a POSIX-ish shell with modern features, Oil (http://www.oilshell.org/) is your best option.

No, it is not. That might be desirable though depending on your perspective or use-case https://elv.sh/ref/language.html

FWIW, XONSH (https://xon.sh - pronounced "konsh") is a POSIX-compliant shell that lets you write everything in Python. (I don't use it, but my cousin is one of its authors.)

My memory of how it works is that it's only POSIX complaint because it does os.system() to a POSIX compliant shell.

It parses some Python syntax on the command line and then uses Python's eval() to handle it. And then for shell parts it calls os.system().

My memory is fuzzy though so I'd be happy for a correction ...

other than seeming inefficient, does doing that have any negative side effects or cause problems somehow?

Well, I'm biased because I'm working on Oil, and I'm sure the xonsh developers are aiming at something very different.

However on top of actually being slow (not just seeming slow) [1], I see it as a huge problem because it makes for an ill-defined language.

The behavior of your shell now depends on whatever /bin/sh and Python interpreter is on your machine. When you change Python versions, are your xonsh commands going to break? What about when you change shell versions? What happens when you copy some xonsh from one machine to another, with possibly different versions? Do they still work?

Some people might say that it's for interactive use, not for programs. But to me the point of shell is that you can seamlessly cut and paste your interactive sessions and now you have full automation. It's an amazing way to quickly build programs.

In other words, shell is the language with the best REPL. And it has the biggest standard library [2]

-----

Oil takes a very principled approach to parsing, e.g. http://www.oilshell.org/blog/tags.html?tag=parsing-shell#par...

while the xonsh approach is just about the most unprincipled approach there is. Again, I think they were aiming at something very different, but to me it falls short of what I want in a shell (and what other people have told me they want.)

-----

[1] I have some experience with running the bash completion scripts that xosnh scrapes: http://www.oilshell.org/blog/2018/10/10.html . It is indeed slow.

[2] https://news.ycombinator.com/item?id=24083764

one could argue that a shell based on python should be especially suitable for turning your commandline into scripts.

i have never tried xonsh but i have used the python repl to interactively build a python script

xonsh approach is just about the most unprincipled approach there is

could you possibly elaborate on that please?

It's not Python though.. it's a tightly coupled mix of Python, shell and other stuff. You would have to extract the Python to actually run it with a Python interpreter, in which case you don't need xonsh at all.

What I mean by unprincipled is that it doesn't parse or define its language. It passes the problem off to other programs (sh and Python).

Is there an Elmish front end?/

Reply on news.ycombinator.com