Elvish – An experimental Unix shell in Go | Hacker News Reader

Elvish – An experimental Unix shell in Go

github.com

221 pointsnetworked75 comments

Author here. Wow, I completely didn't expect this to hit HN this early. I did a talk at FUDCON Beijing 2014 (slides: http://go-talks.appspot.com/github.com/xiaq/elvish-fudcon201...) and warned that "this might eat your ~, so don't post this on HN yet". My original plan was to only post this HN when it's usable, and by "usable" I mean I use it as my login shell on my laptop. (My login shell is still /bin/zsh for now.) But apparently someone else just came across it on the Internet and I should have put up this warning in the README. :)

I'm on the go now, and will come back to add more details and try to respond to questions here.

In one example you pipe a list through some functions, does every function parse the list from a string or is there some protocolly side channel action going on for the typing?

You said you had read about powershell, would be interesting to know if you plan on doing something with datastructures as well as that is its power feature :)

Huh, wow. I was recently thinking about and prototyping a shell seemingly very similar to this. Explicit goals I have in common include:

- Focus on the lispy parts of sh - prefix functions everywhere and a simple, regular syntax. Not that anything with FD redirections as a primitive can get anywhere near S-expressions' simplicity, but shell is naturally more regular than most programming languages (even if bash goes out of its way to be complex).

- Be a real programming language that doesn't make you reach for awk or perl (separate, incompatible environments) to do moderately complex things sanely.

- Emphasize using pipelines rather than 'backwards' function application to naturally string together operations.

- Emphasize lambdas.

- Typed (i.e. non-string) pipes.

- Syntax highlighting.

- Nonzero-exit modeled as exceptions. (I think elvish is doing this, but I haven't reviewed the code in detail.)

Things I want but don't see in the readme include:

- A story for passing typed data around between processes (potentially in different languages) rather than just builtins. I'm not sure exactly what the story should be, but it should exist.

- A somewhat more succinct syntax, with metaprogramming kept in mind.

- A JIT, eventually.

There seems to be a lot more in the first category than the second... give up? But elvish isn't anywhere near done, and I want to do things all my own way for once. I'll keep going, and post my project on HN if it gets anywhere. :)

But to the OP, congratulations on elvish. Hope it gets finished and seriously takes off.

PowerShell does pretty much all the things you mentioned except the lispy part. Actually, I guess it has first class and anonymous function support so if you squint a little you have the lispy part as well.

PowerShell is a great source of inspiration (I went through a PowerShell reference book before I started Elvish), but IMO it's too complex and has too many (avoidable) pitfalls.

I don't know much about PowerShell, although I've heard it's good but somewhat verbose (something I can't stand). I should probably read more about it to see what lessons it has, but of course, as an OS X user, I can't use it.

Xerox Parc Interlisp-D, Smalltalk and Lisp Machines cover all of that and then some.

Their REPLs are more powerful than UNIX shell, you get the whole OS at your disposal using the same language.

The REPLs are graphical, similar to the ideas being brought back by IPython.

Combining objects or functions on the REPLs is Zen compared to the simple UNIX pipes.

> - Nonzero-exit modeled as exceptions. (I think elvish is doing this, but I haven't reviewed the code in detail.)

No really. I have not yet decided how error/exception (http://www.haskell.org/haskellwiki/Error_vs._Exception) handling should work.

I actually like how golang does it - errors with panic/recover, exceptions with return values. But I have not been able to make a decision yet - the situation is a bit more complex than golang. As a result there is no if/else builtin yet, let alone raise and try/except (or panic and recover).

For the record, I am personally usually a fan of making exceptional conditions return values (though I prefer Rust's explicit unwrapping to Go's approach), since in most programs it's good to think at least somewhat carefully about error handling anyway. However, in one-liners, there is no need for proper error handling, and the vast majority of exceptional conditions (e.g. file not found) are caused by mistakes that should be rectified by changing the code. Therefore, in a language that prioritizes one-liners, as I believe any shell should, I believe the traditional exception approach is the best.

> - A story for passing typed data around between processes (potentially in different languages) rather than just builtins. I'm not sure exactly what the story should be, but it should exist.

I haven't thought much about FFI yet. But since in shells you interact with other programs so much, adding FFI would a worthy attempt.

> - A somewhat more succinct syntax, with metaprogramming kept in mind.

Coming up with syntax is actually the hardest thing I find when writing Elvish, since there is much more syntax constraints for a shell. Would be happy to hear advices.

> - A JIT, eventually.

Maybe later.

Remember this?

http://en.wikipedia.org/wiki/Common_Object_Request_Broker_Ar...

By these guys?

http://www.corba.org/

Whatever happened to that?

It's been replaced by dbus. Which does indeed have strict types and other features that might be desirable (multicast?). You raise an interesting idea: A shell that pipes materials around from process to process through dbus, and a set of command line tools that accept input over dbus.

http://en.wikipedia.org/wiki/D-Bus does not work for IPC between computers, AFAIK, only for processes within one machine. CORBA works across machines.

It was terrible on multiple levels and deservedly faded away after a short fad.

And this:

http://google.com/search?q=xml-rpc

Still being used some.

I think the points about types are really interesting. It sounds like a worthy project for that alone.

Does there exist anything like 'type protocols' across languages? Would it be feasible?

Well, there's JSON and variants. Probably a lot simpler than what you had in mind, but if Unix pipes were based on JSON (and had a suitable shell language to work with it), that would already make many tasks easier, like

- Grep the output of ps, without having to count column numbers or do special work to preserve the header line after filtering.

- Search for files matching some conditions, without remembering the weird and unique syntax of find(1) - instead you would write something vaguely along the lines of

    ls | filt {< $1.size 500} {eq $1.type file}

where the stuff after the pipe is the shell language, so theoretically more memorable than

    find -size -500c -type f -maxdepth 1

As a bonus, if you want to cache the output for multiple searches rather than going through the the kernel every time (OS X find is rather slow on huge source trees), you can save the output of ls to a file and keep the rest the same. Good luck doing that with find.

- Handle filenames with spaces and newlines without issues. Simple, but newlines at least are near impossible to solve in the standard shell environment - only a few select tools support null-separated lists. Spaces are easier... unless you want such extravagances as multiple filename columns in a table.

Of course it's not that simple: structured data can sometimes be hard to work with and formatting for display is an issue; Unix's "everything should be plain text" philosophy didn't come out of nowhere.

But I honestly believe that it's time to reexamine that tenet.

Let's take a concrete example: Say you have a programming language L that has a parser that is implemented in lang. A, and an interpreter implemented in lang. B. Now if you want the interpreter (in B) to interpret the program, you need some structured data to come from the parser (in A). Abstract syntax trees (ASTs) are the usual conceptual tool for this. Now assume that the parser outputs an AST that is described by something, like for example an algebraic data type (ADT). Assume that the structure of the ADT conforms to some type protocol. Now the interpreter can implement the ADT in its language (B) and thereby take the ADT directly as input. Now you can use a "typed pipeline" to easily combine these two programs:

> parser directory/file.lang | interpreter

Does this make sense? Would it have some utility?

RPC has to solve the type protocols issue often. Eg Google's protocol buffers.

Protobuf.

Why do we need a new 'real programming language'?

Because it's both a real programming language and a shell, which is rare.

Yes, this is the main reason for me. Where being a shell has two parts:

First, it's universal. Python and Perl and Java all have huge ecosystems, but one of the few places where programs written in all those languages can come together is the shell. By "real programming language" I don't mean some monstrosity with its own packages and whatnot. I just mean a simple principled language to tie things together that doesn't feel like a Turing tarpit.

Second, its highest priority is efficient interactive (REPL) use (although, as a "real language", it should be much better than bash for scripts as well). Some languages have good REPLs, but most don't, and there are important design considerations for them.

For example, if you're banging out a one-liner where you have some data and make a series of transformations on it - thinking through each step in turn - you don't want to have to write functions in reverse order of application, third(second(first(data))). With pipelines, it's forward, first | second | third. In a batch program (in both imperative and functional languages, though more often in imperative), you might avoid this by performing each step as a separate statement, assigning each result to a descriptive variable. But interactively, who has time for variable names? (The elvish author makes a similar argument in the readme.) Yet few languages support this outside of OOP dot syntax, which only works in a subset of cases.

Tangentially related: readers interested in alternatives to bash might be interested in a golang library called Gosh that provides shell-ish dsl, which you can see an example of here: [1].

Author here; I use it regularly to replace bash scripts in system glue code -- using golang channels to pipe data between shell commands is awesome. It's heavily (heavily!) inspired by amoffat's "sh" library [2] which lets one call any shell program as if it were a function.

But, Gosh exists to scratch some itches as a shell scripting alternative. It's nowhere near the full-fledged interactive shell environment Elvish is gunning for. Elvish looks to have a very exciting future :)

[1] https://github.com/polydawn/pogo/blob/master/gosh-demo.go

[2] https://github.com/amoffat/sh/

This looks really cool and promising! I was thinking of a shell along these lines, props to you for writing it.

I do take issue with one of the things you wrote though:

> a more complex program is formed by concatenating simpler programs, hence the term "concatenative programming". Compare this to the functional approach, where constructs are nested instead of connected one after another.

There's nothing about the functional approach that necessitates writing "nested" functions, as you describe. With higher-order functions, you can structure your code in almost any arbitrary way. In particular, haskell's >>= operator has this behavior, and you can easily write an operator like

    x |> f = f x

to facilitate something like

    2 |> addOne |> timesTwo |> show |> reverse |> putStrLn

Or whatever one desires :)

I was actually shown this by another Haskeller once, and I agree that Haskell is a very expressive language allowing for a myriad ways of doing things, including the concatenative paradigm.

But I have always doubted that if the language doesn't explicitly endorse a particular paradigm, you will find it pretty awkward to work with other people. Which is why less expressive languages that emphasize particular ways of doing things still make sense.

Elixir supports that syntax. It's a functional language that runs on the Erlang VM. I found it pretty fun in my experiments, you should check it out: http://elixir-lang.org/

Similarly, Clojure has in its standard library (clojure.core) constructs called 'threading macros' which let you write pipelines just as you would here:

  (-> 2
      addOne
      timesTwo
      str
      reverse
      (apply str)
      println)

UNIX pipes are just a basic version of what is possible when data structures and objects are combined instead.

Which was possible already in Lisp and Smalltalk based OS.

Combining functions raises the experience to another level. :)

Heavy use of backquote might be a problem because it's one of the first keys to drop when keyboards are localised.

+1. On German keyboards for example, the backtick is on a dead key, meaning that it is only printed if you press the spacebar afterwards. Thanks to dead keys, you can e.g. get "é" by typing <´> <e>. But it is a severe annoyance if you actually want to get "`" (<Shift>+<´>, <Space>).

I switched to using a compose key instead of dead keys, because I just end up writing much more code (and english) anyway, so it's more efficient than actually having dead keys in the end.

I'm glad I switched to Colemak way before having to type german umlauts, so I had to learn a few easy combos and keep the rest of the layout unchanged.

I always use the nodeadkeys variants and use compose key to get characters that I don't have on my keyboard, which is many more that the stupid deadkey allows for.

I anticipated that use of backquote might not be popular, but this particular case was unknown to me.

The decision actually arises from avoiding using both single and double quotes for quoting (http://prog21.dadgum.com/172.html), so I just copied golang's string literal syntax.

The shell looks great, good work!

On a side note, I have to say I love README's that look like this. The author did a couple things I really like. 1) They attributed feature ideas to the people they got them from. 2) They listed the good things right along with the bad/lacking. 3) Screenshots.

Hey guys! For anyone who want's to try Elvish without having to deal with golang compilation and such, I just created a terminal.com snapshot. Just register and spin up my elvish container. https://terminal.com/tiny/UtZ8VSgWJL

As it's in development, I will upgrade it again in a couple days.

Hey, that name looks familiar!

Best of luck to xiaq, a fellow fish shell contributor. It's definitely good to see more innovation in the ossified command-line shell space.

Assuming this is planning on using Go's concurrency support, it will be very interesting to see how it deals with the nasty interactions between fork and multithreading.

Thanks ridiculous_fish. I'm still more or less following fish development, and elvish owes a lot to fish.

For now syscall.ForkExec (http://godoc.org/syscall#ForkExec) is sufficient for me. It is written to avoid async-unsafe calls between fork and exec, which is also what fish does IIRC.

Out of curiosity, why do shells need to use fork?

Note: I am not familiar with the implementation techniques behind shells.

The essential function of a shell is to start processes. In Unix and Linux, the usual way to start a new process is to clone yourself (fork), and then have the clone replace itself with a new executable image (exec).

It's kind of roundabout, but the brilliance of this approach lies in what happens between those two calls. There exists process metadata that survives the call to exec, such as where stdout goes, or whether the process is in the foreground. So shells call fork, the clone sets up the metadata for the target process, and then calls exec to start it.

But when a multithreaded program forks, the clone is very limited in what it can do (before exec). In particular, the clone must not acquire a lock that may have been held at the time of fork (which usually rules out heap allocations!). Now say something goes wrong: the clone needs to print an error message, without locking anything. But lots of functions acquire locks internally. How do you know what's safe to call?

fish solves this by providing its own known-safe implementations of printf() and friends, and being careful to only call those after fork. Go solves this by disallowing any user-code between fork and exec. Instead it provides a single posix_spawn-like entry point called ForkExec, and does some black magic (like raw syscalls - see https://code.google.com/p/go/source/browse/src/pkg/syscall/e... ) in between the underlying fork and exec calls.

My hunch is that a shell written in Go will eventually bump up against the limitations of ForkExec. Happily Go has a strong FFI, so you can hopefully implement this stuff in C, if it comes to that!

One of the most frequent things you do in shells is invoking external programs. Unlike in Windows where a CreateProcess does the invocation, in Unix systems this is (traditionally) achieved by a fork/exec combo. See http://en.wikipedia.org/wiki/Fork-exec

If I type 'mplayer ' and press tab in a folder with only one video file, but multiple text files, does this shell autocomplete the media file? It's pretty frustrating that even the most basic stuff like these are not enabled by default on your average Linux distribution. I wonder if everyone simply stopped developing unix shells.

I'm aware that one can install support for proper autocompletion by installing additional stuff. That shouldn't be required, smart autocompletion should work out of the box. I don't want to configure and/or install stuff for every single application that I use.

Bash Completion[1] scripts are what provide this feature; perhaps fewer Linux distributions are including them. I recommend keeping them in your dotfiles[2] and loading them in your .bashrc[3]. This is nice because Bash Completion operates by filename, meaning you can take any project's bash completion script and install it at `$bash_completion/completions/script-name`.

[1] http://bash-completion.alioth.debian.org/

[2] https://github.com/oinksoft/dotfiles/tree/master/lib/bash-co...

[3] https://github.com/oinksoft/dotfiles/blob/master/bashrc#L22-...

There is only very primitive file name completion for now, but programmable completion is planned.

I've been tinkering about with writing my own shell at home, as an excerise in learning more about unix etc but bumped up against the limitation in that proper job control is currently impossible to do from within go (off the top of my head it was something to do with the inability to set the process group correctly for forkexeced processes). How did the author get around this, or does Elvish not have the ability to put processes into and out of the background?

It would be nice if

    put 1 2 3 4 5 | filter {|x| > $x 2} | map {|x| * 2 $x}

could be written just as

    put 1 2 3 4 5 | filter > $_ 2 | map * 2 $_

can you express that grammar in ebnf? I think the parser would need to be at least ll(C) where C is bigger than 1.

Would the presence of $_ denote a lambda? how would nested lambda's work?

Why not:

    put 1 2 3 4 5 | filter -> > $1 2 | map -> * 2 $1

I think that would be easier to parse, and not much typing overhead, also $_ is a weird thing.

This is equivalent to the following bash incantation?:

    cat <<eof\
     | while read a;do if [[ $a -gt 2 ]];then echo $a;fi; done \
     |while read b;do echo $((b*2));done
    1
    2
    3
    4
    5
    eof

(Better expressed with seq):

    seq 5 \
    | while read a;do if [[ $a -gt 2 ]];then echo $a;fi; done \
    | while read b;do echo $((b*2));done

But we can do (a little) better:

    # Ok, this is a bit odd, the filter is actually inverse...
    filter()
    {
        while read it
        do
          #NOT SAFE!!
          if eval ${*}
          then
            echo "${it}"
          fi
        done
    }

    map()
    {
        while read it
        do
          #NOT SAFE!!
          eval ${*}
          echo "${it}"
        done
    }

    #Valid bash:
    seq 5 \
    | filter test '$it' -gt 2 \
    | map 'it=$((it*2))'

Sorry, didn't mean to turn this into stack overflow, the "simple" syntax just tickled me...

Some languages have a sort-of default variable name for cases like this, namely "it":

> put 1 2 3 4 5 | filter > $it 2 | map * 2 $it

As zokier was presumably pointing out, other languages have a different sort-of-default variable name, namely `$_` (http://c2.com/cgi/wiki?DollarUnderscore). :-)

Xerox PARC environments and their derivatives keep being partially reinvented.

I really wonder how computing would look like if those systems had succeed in the market, instead of UNIX.

Hmmm...it's not building for me:

  $ go get github.com/xiaq/elvish
  # github.com/xiaq/elvish/edit/tty
  dev/go/src/github.com/xiaq/elvish/edit/tty/termios.go:20:   undefined: syscall.TCGETS
  dev/go/src/github.com/xiaq/elvish/edit/tty/termios.go:24: undefined: syscall.TCSETS
  dev/go/src/github.com/xiaq/elvish/edit/tty/termios.go:49:   cannot use &term.Lflag (type *uint64) as type *uint32 in   argument to setFlag
  dev/go/src/github.com/xiaq/elvish/edit/tty/termios.go:53:   cannot use &term.Lflag (type *uint64) as type *uint32 in   argument to setFlag
  # github.com/xiaq/elvish/sys
  dev/go/src/github.com/xiaq/elvish/sys/select.go:66: not   enough arguments to return

I opened up an issue for you: https://github.com/xiaq/elvish/issues/16

I think this is because you are on an older version of Go. Try it with Go 1.3

I am using Go 1.3.

  $ go version
  go version go1.3 darwin/amd64

It builds for me with 1.3 (linux/amd64) on Ubuntu, so I'm guessing it somehow works on Linux but not Mac.

EDIT: Specifically, the first two lines are complaining about an undefined syscall and the next two are complaining about something terminal-related being 64-bit where 32-bit is expected, so those certainly seem like plausible things that would differ across OSes.

The syscall package varies by platform, and TCGETS, for instance, is defined for Linux-

http://golang.org/src/pkg/syscall/types_linux.go

-but not for Darwin-

http://golang.org/src/pkg/syscall/types_darwin.go

So it seems somewhat Linux tied.

Clojure has -> specifically to facilitate writing pipeline code that doesn't read backwards.

Or, in this case, ->>:

  (def lols (->> strs
                 (filter #(re-find #"LOL" %))
                 (map upper-case)
                 sort))

How I read it: take strs, filter the strings that contain "LOL", turn them into upper case and then sort them. Basically it reads the same as the pipeline example in the README.

Right – though for this example, `->>` (http://grimoire.arrdem.com/1.6.0/clojure.core/DASH__GT__GT/) works better than `->`. I created a GitHub issue to tell the creator about it: https://github.com/xiaq/elvish/issues/17.

(Also see my reply to thinkpad20.) I'm not sure about Haskell, but Clojure's -> is still a serial construct (functions invoked one after another), but pipeline is parallel.

Many alternative shells popping up recently. Someone already mentioned the Fish shell [1] in this thread. There is also Xiki [2] and its recent Kickstarter campaign [3].

[1] http://fishshell.com/ [2] http://xiki.org/ [3] https://www.kickstarter.com/projects/xiki/xiki-the-command-r...

This is awesome! Necessity is the mother of invention, and all that. The idea of using lispy syntax while dropping the requirement for the outermost pair of parentheses is definitely a neat idea - I'll try the shell out in practice to see how much easier it makes usage, but it seems like it would allow for 99% of the consistency of Lisp/Scheme less 20% of the annoyance for general shell usage.

Looking forward to what's in store.

LispWorks:

    CL-USER 7 > + (sin 1) (cos 2)
    0.4253241

I get this error:

clang: error: no such file or directory: 'libgcc.a'

I really like in-shell floating point operations. But this:

    > / 1 0
   +Inf

Is not the correct answer - even as a limit - since the limit of 1 / x (when x -> 0) can be +Inf or -Inf.

The 0 is a positive one.

    > / 1 -0
    -Inf

BTW this is just IEEE floating point.

Ah yes indeed [1], I didn't know.

[1] http://en.wikipedia.org/wiki/IEEE_floating_point#Exception_h...

So will it work with a gccgo?

"It attempts to prove that a shell language can be a handy interface to the operating system and a decent programming language at the same time; Many existing shells recognize the former but blatantly ignore the latter."

Wasn't that the reasoning behind csh[1]? (She scripts csh by the c source). Did you have a look at plan 9's rc[2]?

I welcome attempts at new/better shells -- doesn't look like elvish will be my saviour (partly due to the heavy use of backticks) -- but there is always need for fresh blood in the battle for the terminal.

I suppose that "vi keybindings that makes sense" and "a programmable line editor" is meant to indicate that rlwrap isn't good enough? I must say, after a few years (has it really been years) of "set editing-mode vi" in .inputrc, I actually think it works kind of nice with (plain) bash. I suppose there's room for improvement in terms of history editing etc... But either due to lack of imagination or force of habit, I've never really felt a pull towards zsh (or fish, or other "improved" shells). But playing around with ipython and/or Conque for vim has made me consider looking for greener shells than bash.

The secret, I think, is to avoid doing to much in the shell, and rather try to subtly improve on the "simple programs build complex pipes"-idea. I actually think some rethinking of core command line tools (cat/tac/tee, grep/sort/uniq, seq etc) might be a better investment than "better syntax".

Not that better syntax is a bad idea -- a "strict" subset of modern bash would be good, with saner handling of words/expansion/substitution -- essentially defaulting to proper checking for empty variables ("${might_be_empty}a" == "a") (but isn't [[ -z "${var} ]] always better anyway..?), always defaulting to "${var}" rather than $var, ${var}, and preferring var="$(some_command_that_outputs)" to var=`dirty backtick command`...

Essentially getting rid of all the crazy old cruft that's needed for backwards compatibility, and defaulting to sane, modern versions (it's what's hard about scripting (especially posix [k|b])sh -- there are 5 wrong ways to do everything, 2 mostly right and 1 perfect -- but which is perfect often depends on context...).

[1] I'm not a csh-fan, for some reasons, see: http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/ But mostly I'm just grumpy and conservative, and having mostly figured out how to properly get things right in ksh/bash, I stubbornly refuse to use something else ;-)

[2] http://swtch.com/plan9port/man/man1/rc.html

Ironically, Rob Pike eschews big shell scripts.

If you want a programming language, you know where to find one.

Yep, I just read that as Elvis with Sean Connery's accent... Nice work though.

> It attempts to exploit a facility Shell programmers are very familiar with, but virtually unknown to other programmers - the pipeline.

Really?

It seems that Unix pipes are the go-to example of what you might call pipeline programming, as if it originated there or because every programmer is a shell-programmer first and foremost. But I'm not sure that this is such a secret technique exclusive to shell programmers - object-oriented languages can and does seem to like to use "fluent interfaces", I think its called, which has the same pipelining style. Functional programmers are able to and probably find it convenient to use a "pipeline style" on longer expressions that are essentially long chains of function application or function composition - they just have to flip the order of the operators for function application and function composition, respectively. This is possible in languages like Haskell, and I think it is even pretty idiomatic in F#.

Unrelated, but I would like to note that you shouldn't use subpixel anti-aliasing in text in images, especially ones that will be shown by a web-browser. You never know whether the user's screen is rotated or not; if it has three subpixels per pixel, or if it's pentile or similar; or if it is high-DPI and therefore the image is zoomed; and even if none of those are the case, a lot of people dislike the coloured fringes.

Good to know that, thanks. I have captured and pushed the screenshots with subpixel turned off.

Reply on news.ycombinator.com