Starting off in Rust: Trying to write a shell | Hacker News Reader

Starting off in Rust: Trying to write a shell

blog.achernya.com

88 pointsmadars19 comments

> The POSIX shell grammar is ridiculously complicated, and would require the use of a parser-generator to reasonably implement.

Wat? Admittedly Bourne shell syntax is ad-hoc, glitchy to implement and difficult to specify. But it's absolutely not something that requires a parser generator. It was always and remains a string replacement language without deep structure.

Bourne shell syntax is inspired by Algol 68¹, of which Bourne was a fan — the shell itself² was written in a highly idiosyncratic dialect of C, using the preprocessor to provide an Algol68ish syntax for control structures:

    WHILE i--
    DO  t=trapcom[i];
        IF t==0 ORF *t
        THEN clrsig(i);
        FI
        trapflg[i]=0;
    OD

The parser³ is straightforward recursive-descent in 400 lines including comments & blanks (the whole shell is about 4000 lines of code).

¹ http://en.wikipedia.org/wiki/ALGOL_68

² http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/...

³ http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/...

Shame the blog is so horrible because I'd love to read the text and enjoy a hypertext document or two.

I can't fathom why Google doesn't streamline that crap so that it doesn't depend on scripts from two or three separate domains and will properly fallback render to something decent without any scripts.

Is it really true that rust doesn't support i/o redirection as the article states? It looks like the apis are there: https://doc.rust-lang.org/nightly/std/process/struct.Command...

In addition to null() you can use piped().

Does a shell really exercise the use of data structures? To my mind it seems there's very little state, and all of it in substantially simpler formats than even e.g. JSON.

Yeah, probably not. In any program, if you don't care about performance, you essentially don't need anything more than hashes and arrays (what Python/Perl/Ruby/JS provide).

In a shell you don't care about runtime performance because all the cost is in the invoked programs. Ironically you probably care about parsing performance more than execution performance, because parsing happens at startup and can cause latency.

You will have a stack since I think shells have some form of scope, but that's pretty much an array.

> In a shell you don't care about runtime performance because all the cost is in the invoked programs.

This is not strictly true. In a reasonable shell many commands will be builtin, because invoking processes for, say, echo, is slow.

What you said is true, but it doesn't make what I said untrue.

It could be that 99% of the statements are executed by shell builtins. But for essentially any shell script I've seen, speeding those up by 10x won't make any difference, because the real work is invoking compilers, linkers, invoking curl, tar, gzip, etc. As well as simple process creation overhead.

Agreed - when I play with a language I normally like to building something like a simple Trie implementation to get a feel for the collections library.

I guess a shell is a pretty good learning project though, as you can just expand its scope to touch upon most of a language's API surface area as you go:

- Want to see how a language handles IO? cd, ls, mkdir and friends will give you a pretty good introduction.

- Want to look at the http lib? A simple wget clone.

Rust is supposed to be a language for systems programming, and a shell seems like a good exercise for that.

Is it? I understood it was a language for Mozilla (i.e. with the browser as a core use case) - so for very high-performance applications (those that need so much control of memory management that even Haskell is unacceptable) rather than systems per se. No?

Seems like pretty nebulous territory to me (which I think means it's probably not worth splitting hairs over it). In any case, rust-lang.org says in big giant letters: "Rust is a systems programming language ..."

Writing a browser is very much a systems programming task, including careful memory management, access to numerous platform features, implementation of a high-performance JIT and language runtime...

It's not really systems programming in the sense that I understand it (or in e.g. the wikipedia article). More to the point, I think a browser involves a lot more manipulation of complex data structures - even the basic DOM is more complex than anything you'd use in a shell - and this is an area in which I expect Rust to shine. So I don't think a shell provides the best opportunity to appreciate Rust's strengths.

Good project, to write a shell in Rust. There are other efforts underway to rewrite the low-level UNIX utilities. An all-Rust BusyBox is probably not far off.

The author's problems with language and library stability, though, comparable to my experience with Rust. The libraries aren't there yet.

I've been trying to write a program that uses RSS feeds. There's no RSS feed reader library, so I have to write that. RSS requires an XML parsing library, an HTTP library, and an ISO date parser. There are several choices for an XML parsing library, and one of them worked and produced a good tree. It doesn't have any functions for finding elements in the tree, though, so I had to write those.

There are at least three HTTP libraries. One is obsolete, one isn't done yet, and one, "hyper" is usable. Or it was, until last week, when the "io" library had a major redesign. Using "old_io"? That's so last week. Right now, the "hyper" HTTP library is broken, due to some upstream problems with the SSL and cookie libraries being broken by the change at the "io" level. The appropriate issue tickets have been opened and many people are working hard on fixing everything that does I/O to do it the new way.

ISO date parsing wasn't implemented for the date/time module, so I wrote that, forked the date/time module, and submitted a pull request. The author of the date/time library rewrote the code and added it, and now the stock "chrono" module does the date/time formats needed for RSS and Atom feeds.

Compiling my own code, I now get messages like

    rssread.rs:262:23: 262:25 warning: obsolete syntax: []
    rssread.rs:262     p.feed_str(&string[]);    
    note: write `[..]` instead

The syntax for a slice that includes everything changed since last week.

The documentation for all these changes runs behind the code, of course, so you have to read blogs, issues, and pull requests to figure out what the language does this week.

Daily Rust development cycle: "sh updaterust.sh", then for each project "cargo clean; cargo update; cargo build; cargo test", then fix what's now broken.

I can understand what the Rust developers are going through. Their public pronouncements [1][2], though, create the illusion of a higher degree of stability than is actually the case. If you have to do any real work, I'd hold off on Rust until the 1.0 release, and maybe a few months after that.

Rust is going to be an important language, but it's not ready for prime time yet.

[1] http://blog.rust-lang.org/2015/01/09/Rust-1.0-alpha.html [2] http://blog.rust-lang.org/2015/02/20/Rust-1.0-alpha2.html

That's your perspective. Here's mine. I've been keeping a number of crates updated for Rust nightly for almost a year now (with a lot of help from the Rust community), and the pace of breaking changes has slowed down dramatically. IO reform has been loudly talked about for quite some time now, and it really should be the last big breaking change. Every time I look at the Rust docs, I see more and more green stable indicators. All of the Rust announcements reflect my experience pretty accurately.

To read this without JavaScript, you can get the post text from the (current) top of the RSS feed:

http://blog.achernya.com/feeds/posts/default

The source code of the (unfinished) shell is at:

https://github.com/achernya/xv6-shell.rs/blob/master/src/mai...

For these Blogger templates, I just pull the relevant page from Google Cache. E.g.

http://webcache.googleusercontent.com/search?q=cache:http%3A...

Reply on news.ycombinator.com