Building your own shell using Rust | Hacker News Reader

Building your own shell using Rust

joshmcguigan.com

216 pointsboyter52 comments

Hi all, author here. I'm not sure of the rules around self promotion so I want to point out that I didn't submit this, but I'm happy to see that others are enjoying it.

I wrote this blog post to document my process of building a simple shell using Rust. I learned a lot about how shells, terminal emulators, and the OS interact by going through this exercise, so I hope this is useful to others as well.

Feel free to reach out if you have any questions.

For future reference, it's fine to submit your own stuff as long as you don't overdo it. For things that can be tried out, there's also a special Show HN category: https://news.ycombinator.com/show

Tooting your own horn is a foundational block of HN and very welcome!

If you have open source projects, then I would like to join

I've linked my GitHub profile below. I really enjoy social coding, so feel free to open pull requests or issues on any of my repos.

https://github.com/JoshMcguigan

I've never used Rust, but looking at the code, and reading https://doc.rust-lang.org/std/process/struct.Command.html , is std::process:Command just farming out to bash? I ask, because the examples don't specify a full path, so that implies something else is doing the PATH search, like bash. Because of this, and the semantics, it looks like it's equivalent to a system() function in other languages.

If you really want job control, a path, environmental variables, and all the things that make a shell a shell, you have to be using exec() or execvp() https://doc.rust-lang.org/std/process/struct.Command.html#me... . Otherwise all you've done is wrap an existing shell that does all the work for you, and that was an immediate fail when we had to a make a shell in my undergrad systems course.

An easy check is the use the command "echo $SHELL". If that spits out anything besides literally "$SHELL", you're farming out.

All that said, Rust's exec() is listed a unix specific, so if you're on Windows, I guess you're stuck.

I'm not confident enough to say this for sure, but I think you are off base.

I created a demo in the Rust playground [0] which shows that there is no implicit environment variable expansion. Also, the source for `std::process:Command` [1] looks to be calling into libc to perform a fork/exec.

That said, this is rather new territory for me, so let me know if I'm missing something here.

[0] : https://play.rust-lang.org/?version=stable&mode=debug&editio... [1] : https://github.com/rust-lang/rust/blob/master/src/libstd/sys...

It will use `execvp` (which automatically reads the PATH variable, no shell required) if you use features that require it (like `before_exec` which runs a block of code between `fork` and `exec), but it generally prefers `posix_spawnp` (Which also uses PATH). The relevant code is here: https://github.com/rust-lang/rust/blob/7d3b9b1640611c52eb949....

All of this is of course Unix specific, on Windows it uses `CreateProcess`.

If you or anyone else would like to file a bug, that’d be great: we should make this more clear in the docs. I’m about to jump on a plane or I’d just do it; a bug will make sure I get back around to it eventually.

Why would the docs need to be updated? Is it assumed that a command would be executing a bash or other shell process?

If something isn’t clear, it deserves an update.

This could be by adding some text to clarify that this doesn’t use a shell, or maybe changing the examples to not imply one is used. It just depends.

The internal implementation of Command is using libc::execvp() on unix and CreateProcessW on windows. In windows case, it's searching the executable in the PATH manually first to match the semantics of unix.

If you look at the examples in the first link you posted, you can see to farm out to an existing shell, they're explicitly invoking "sh". There would be no reason to do that if Command always farmed out to another shell.

If anyone is interested in developing a shell even further, checkout rustyline[0]. It makes it very simple to add keyboard shortcuts (Ctrl-C), completion, and even history. There's also liner[1], which is part of the redox-os project, which also has a lot of the same features.

[0]: https://github.com/kkawakam/rustyline [1]: https://github.com/redox-os/liner

I'd second the recommendation for rustyline. I played around with it a bit while reviewing the rush[0] project, another shell implemented in Rust for learning purposes.

[0]: https://github.com/psinghal20/rush

I second this greatly! Too many people roll their own line editors and always mess up the readline (emacs) shortcuts.

I rolled my own and didn't even bother with emacs shortcuts (got vim ones though, as that's what I'm familiar with).

However I'm not adverse adding emacs shortcuts if anyone can recommend a good guide for which keys do what.

Bash shortcut guides get most of the useful ones. https://github.com/fliptheweb/bash-shortcuts-cheat-sheet

Useful link (thank you) but to be honest I was after a more complete list because nothing breaks usability more than something being familiar but not behaving quite the same exactly. But I'll stop being a lazy arse and Duck Duck Go the answer myself :)

ctrl+c (as well as ctrl+\ and ctrl+z) is a little different from other shortcuts because they're captured by your TTY rather than your running application - though you can catch the interrupt that ctrl+c et al sends.

Lesser known fact: you can also redefine which keys you want to raise SIGINT, SIGQUIT and SIGTSTP with.

An example of a more fully featured shell written in rust is Ion. https://gitlab.redox-os.org/redox-os/ion

It is inspired by fish

I'm currently writing a shell in Rust, and have written a post about it. If you find this interesting you may also find my post interesting.

https://nixpulvis.com/ramblings/2018-10-15-building-a-shell-...

Thanks for sharing, your posts are very well written.

I really like the idea of syntax highlighting before the user presses enter.

Any reason you are writing your own posix parser and readline implementation, rather than using existing Rust crates?

Partially to learn how, partially to maintain a consistent style, and partially to see where we can improve.

As for the POSIX parser specifically, I love grammars, and want to support multiple, and building one in LALRPOP is t too bad. Also I looked at conch-parser, but didn't want to use its AST, and decided to go this route.

I have been using the termion crate, and may have another PR to contribute. I have somewhat ambitious goals for the readline like library though.

As I was thinking about an improved user interface for the shell, it seemed like at some point you'd hit the limit of what is reasonable given the current split implementation between terminal emulator and shell. Do you think you'll get to a point where you'll need to build an all-in-one solution?

It's interesting you mention that. I'm (slightly) involved with the Alacritty project, and generally like the layer of abstraction here. If not only to support the existing status quo, but also as it leaves a lot of the platform specific features for the terminal.

ANSI, and the other standards are large and somewhat unwieldy though, and there are times it would be nice to do away with it all.

As for features, there's a number of things that you could argue for being implemented in either place. One feature I feel is a good fit for the shell for example is saving and being able to recall and/or pipe previous commands, while copy paste might be implemented in both.

Any other recommended article in making your own shell in other language?

https://github.com/danistefanovic/build-your-own-x#build-you... ?

Reply on news.ycombinator.com