A Bourne-style shell built from scratch in 35 minutes [video] | Hacker News Reader

A Bourne-style shell built from scratch in 35 minutes [video]

destroyallsoftware.com

197 pointsgary_bernhardt64 comments

On this note, I need to build my own Terminal multiplexer - does anyone have a favorite primer of theirs for all the escape codes? Not only do I need to use them myself, but I need to handle them of child processes to restrict term sizes etc.

I can Google like ya'll - I'm just asking if there are any favorite resources :)

Thanks

There are no such things as “the” escape codes. Any terminal could have any codes. You would have to look at the TERM environment variable and use codes appropriate for that terminal. There are, of course, libraries to do this for you, such as terminfo and ncurses. But never output raw escape codes yourself - all the world is not a VT100 or Linux terminal.

I think the original poster does not want to output escape codes, he wants to build a terminal that will accept escape codes like a xterm. This way, he won't need to create a new terminal type with all the terminfo or ncurses configuration.

The two other terminal multiplexers I know of are GNU screen and tmux. GNU screen defines its own “screen” terminal type in terminfo, and tmux shamelessly re-uses it and claims to be “screen-256color” (screen with 256 colors). So I don’t think there is any good way to avoid it.

I just assumed it's what someone who'd actually thought about the problem did for performance.

>"shamelessly re-uses it and claims to be “screen-256color”

Could you elaborate on this? Tmux uses the screen terminfo entry? Is it not actually 256?

If I set my terminal type to something basic like vt102, and then start "tmux" and run "echo $TERM", I get "screen-256color". Also, the tmux(1) manual states:

The TERM environment variable must be set to “screen” for all programs running inside tmux. New windows will automatically have “TERM=screen” added to their environment, but care must be taken not to reset this in shell start-up files.

So, yes, Tmux uses the GNU screen terminfo entry.

I did some research a while back and what i recall is that Xterm is the Gold Standard, supporting more escape sequences than anything else. So, if you can come even close to what Xterm supports you're good. PDF of xterm control sequences https://www.x.org/docs/xterm/ctlseqs.pdf This site is good to for explaining xterm's behavior: http://invisible-island.net/xterm/ and how it came to be what it is and support what it does.

I've seen claims that current command line tools are standardizing around the ECMA-48/ISO 6429/ANSI X3.64 escape code protocol

The xterm document pointed to elsewhere in the thread ( http://invisible-island.net/xterm/ctlseqs/ctlseqs.html ) is the canonical reference, but I'd suggest using something like libvterm rather than writing that part yourself.

Not strictly answering your question, but years ago I worked on a job which comprised a group of us sitting in a room, each with a VT100 connected to a Unix box in the corner, cranking out C. We weren't massively overworked, so one clever bugger wrote his own terminal multiplexer on the side, enabling him to flip between sessions using function keys. Another guy wrote a shell - in Bourne shell. It had, errm, a few issues, but, crucially, gave us command line editing and history (that was otherwise lacking in the vanilla sh we had, and none of us fancied csh much).

My 2nd year college project (UK college not US very different things) was a VT100 emulator written in Delphi.

In hindsight I should have done what the rest of the class did and took the Blackjack option but there was the option to do our own project instead, I demonstrated it worked by connecting it to my Linux laptop (this was '98, I was a strange 18yo).

I was into LambdaMoo and mtrek at the time and I wanted to understand how they worked better :).

why do you want to build your own multiplexer? is it just to learn the tech, or do you find screen/tmux aren't what you're looking for? (if the latter, what's lacking?)

i'm not trying to say it's not worth building your own, i'm just curious.

Sorry, missed your comment.

Yes, to learn the tech - but also to toy with ideas. I've been using Kakoune recently after years of Vim and while I enjoy it, it makes me wonder many things. After much speculation to myself, I decided to try implementing my ideas. I mainly expect them to fail, in the UX department at least, but I have to use it to really flesh it out.

This may sound more editor oriented, and you'd be right, but for a simplest explanation a self made multiplexer like tmux/etc will be involved.

cool! good luck.

Then 35 weeks to get POSIX job control, pipes, handling signals and whatnot. 35 years to find the last bug.

Probably a bit more than 35 weeks, if that POSIX control is going to be tested across multiple UNIX variants. :)

Not that I think writing a feature-complete shell is “easy”, but pipes, specifically, were the main feature of this shell implementation (besides reading, forking, and executing).

The video isn't advertising this as a shell you'd use in production.

It's an interesting resource to learn how shells work and are implemented. It doesn't cover absolutely everything, but it was never meant too.

It supports (1) running commands with arbitrarily many arguments

Wouldn't the size of the arguments be limited by sysconf(_SC_ARG_MAX) as explained here? http://man7.org/linux/man-pages/man2/execve.2.html

PS: Is the code available somewhere? I can't play the video on Firefox on CentOS 6.

The Bourne Shell was, in my opinion, the worst of Robert Ludlum's books.

By the way, IMHO much of the ugliness of shells comes from the requirement that simple filenames are to be used without additional escape characters. I think we should ditch that requirement.

My fantasy crosscutting change is outlawing spaces in filenames, at the filesystem level!

my fantasy is they design a shell language where it's impossible to write anything that fails if there are spaces in filenames

That's much better than what the parent wants. Almost always whenever I write any script almost 25% of the time is spent ensuring it doesn't break due to spaces or special unescaped characters.

You already have that. Just write your script in one of the countless sane languages that are #!-able.

until your script needs to launch/spawn another app/script/command/program.

Doesn't necessarily have to be a major hassle. With my homemade Lisp, here's a quick and dirty shorthand for running external programs (ultimately boiling down to fork+exec):

    (extend-parser "!"
                   (mac (cmd) `(apply run (map to-string ',cmd))))

    !(sed "s/foo/bar/" README)

and uppercase.

Only typing the man page of the Bourne shell will take at least a few days.

Not to mention the time it takes to think it up, i.e. make the design decisions.

"That's not writing, that's typing", as someone once put it.

There is more to writing a shell than simply reading commands and executing them after fork()ing. TTY handling must be done right¹ and interrupt and signal handling is rather tricky to get right as well².

(Not to mention, of course, that a Unix shell is very cumbersome to use without pipes and redirection. EDIT: Pipes are implemented, but not redirection?)

1. http://www.linusakesson.net/programming/tty/

2. https://www.cons.org/cracauer/sigint.html

I’m not sure if you watched the video or read its description, but pipes are the focus of this implementation. He describes pipes as “the core of the UNIX shell” in the video. I suppose that is somewhat up for debate, but building complex tasks out of composable pieces is certainly up there among UNIX’s design goals, and pipes are a big part of that. OTOH, his implementation does not seem to cover output redirection à la `exec` (file-based, file-descriptor-based, or named-pipe).

I read the description, but must have missed the part about pipes; oops.

If the implementation done in less than an hour is not featureful enough for you, obviously you don't have to use it. Doesn't reflect the value in this type of post on HN.

If it was insufficiently pedagogical, or failed to give people the impression that shells are just software and They Can Do It Too, then I would understand the critique more, as those are clearly the goals.

I was trying to notify readers that the description is rather inaccurate; The HN title claims “Bourne-style shell”, and the article claims “shell in the style of the Bourne shell, Bash, zsh, etc.”, but a working shell actually requires TTY and signal handling; they’re not really optional features.

Sure, if all you’re doing is implementing a program with a command line interface like ftp(1), kermit(1), sftp(1), etc. that do not run arbitrary subprocesses, and do not need to handle being arbitrarily interrupted, then sure, implement away. I would suggest the Python “cmd” library as a good way to get started quickly. But for a login shell, used to run other programs, there are more complex considerations.

I'm surprised there are no comments about the video itself.

How was this video made? Imagine trying to make it. What would you do? You need to write the code, but do you do it before or after you write the script? Or do you write the script after you write up the code so you know how to produce it?

Everything flows so smoothly. It's mesmerizing to watch. It's like code appears as fast as he can talk. But in a natural way, where it seems like it's typed as he's talking. But he certainly isn't typing it in realtime.

I'd like to make videos like this too, so I'd like to learn more about how to produce one.

That typing is in realtime! I rehearse screencasts until they flow naturally like you see here; there's no content editing and no script. I basically record the whole thing from beginning to end, over and over, until it doesn't change significantly from run to run and I don't feel it catching anywhere.

I do edit out pauses to breathe, to drink water, sometimes to double-check my thinking about what to do next, and occasionally to open a second terminal and check something.

Destroy All Software is six years old, so I've had a lot of practice. Most DAS screencasts require a couple hours of initial silent prep, then around 10 full runs from beginning to end. When it starts to flow well (run #5, give or take), I fire up ScreenFlow and start doing each run as if it might be the final published version (because it might actually be). With these longer "from scratch" screencasts, I only make one or two attempts per day; by then my throat and brain are both pretty tired.

I also live stream these on Twitch before making the final version, so you can see the sausage being made if you like! It's basically the same as what you saw in this video, but the pauses are there, you get to hear me become confused sometimes and comment on my mistakes, and there's a bit of Q&A afterward. The schedule isn't fixed, but I try to tweet with a couple hours' notice (https://twitter.com/garybernhardt).

Thanks for the candid transparency, it's very appreciated. Someone with much more hubris might claim to have recorded it in one shot, first attempt, as some ridiculous technical interviewers seem to expect you to do.

I think it really speaks to how software development happens. The premeditated thinking takes up most of the time, and typing it out is often the easiest part when the architecture is crystal clear in the mind. Combining that with succinct explanations might be like putting both hands together while playing piano.

I am not downplaying the difficulty of what you're doing at all, just appreciating the process. Thanks for the material!

I learned a whole lot from your DAS screencasts! They came along at a time when I was leaving my first software dev job where I had spent a decade doing things the way they had always been done there. Things like automated testing, MVC, and for the most part web development were all completely new to me. I bet I watched most of them more times than you ran through them making them. I would watch them over and over until I understood the concept, then find places at work to apply what you taught me. Thanks for making them.

Do you still do twitch sessions ? I only saw one tweet a few weeks ago (which was instructive).

I saw Gary do it live and in person. You can bet your ass it's live.

I've never seen anybody else who is able to do that. It's truly mind blowing.

Such negative Nellies. This is why I don't publish anymore. You try to teach people something fun, and show people you can do quite a lot with very little. But it's nothing but criticism, and none of it constructive.

Totally get your point, but not being all that you are out of fear for consequences will never work out. Whiners and trolls are way overrepresented, it's a way of life for them. But for every confused individual there's someone who's been waiting to hear your voice, someone who's just as sick and tired of the same old as you are. This is your space as much as anyones, our space; as long as we're willing to claim it.

At the time of this comment, this post has 70 votes and 25 comments. 70 people that liked it enough to upvote it, and not all of these comments are negative. I know absolutely nothing about you, but if this HN thread is representative of why you don't publish anymore, might I suggest a thicker skin.

It depends on where you look. Hacker News is pathologically tilted toward negative sentiment, usually in the form of look-how-smart-I-am. I also announce these screencasts on Twitter, where the the response is universally positive (in six years, I can't remember a single rudely negative comment about a Destroy All Software screencast). Not that Twitter lacks its own problems, of course!

Was expecting more guns, car chases, exotic city backgrounds, and slightly-above-average women from the new Bourne shell. Turns out it's just a command line interpreter. Two stars.

Funny, but the shell's namesake is pretty badass himself https://en.wikipedia.org/wiki/Stephen_R._Bourne

“[…] at one time Bell Labs had two Stephen R. Bournes with offices along the same hallway.”¹

— Eric Allman

1. http://www.sendmail.org/~ca/email/README.cf.8.8.html

Embarrassing that I use bash everyday and didn't know about Stephen Bourne. Shameful.

I also just learned Bash was developed by Brian Fox after Stallman became dissatisfied with progress on it in 1988 as a free software replacement for the Bourne shell.

We don't celebrate these guys enough, I wish there was more recognition, more of a public face and profile for all these pioneers.

I don't really think there's any call for such an objectification of women. The joke would have been just fine without it.

It wasn't objectification of women, it was a reference to a Bourne (movie series) continuation.

I'm not sure how to respond to this. I can't tell if you read both sentences of my previous comment. My point was that the Bourne reference would have been funny without the objectification of women by the inclusion of the phrase "and slightly-above-average women" in the comment at the top of the thread.

Will this video eventually be paywall'd as well?

The video has a download button (right click save as also works) if you want to save it for later.

I'm impressed that destroyallsoftware is able to put their screencast library behind a $30/mo paywall. Am I the only one who thinks that's high?

No I agree, but then again that's a dollar a day and apparently he could buy a house in Seattle out of DAS so.. there must be enough people that are fine with that price to make a living.

I think, for the quality, it's on the lower end. Most such services that I subscribe to (or have in the past) are $20-30 per month. The quality of both the production and content is easily worth the money. Even more so now that there are 4 seasons of some really high-quality screencasts from the past.

The value I've personally got out of DAS is incredible.

When you set up a paywall, you cut the number of people who will consume your content by a ton, and then you have to set the price pretty high to make a living.

The reverse being that with a ton of people that watch your free content you will probably make less than what the author currently makes.

But 'ls' already has built-in sorting.

I was so disappointed that they chose Ruby. I would have been impressed if they had chosen bare C and still manage to build it under 35 minutes.

stephen brennan wrote a great article [1] on how to write a shell in C. it's a little more simply than DAS' implementation, but it's a fantastic read.

[1] https://brennan.io/2015/01/16/write-a-shell-in-c/

For my system programming class we had to write a Unix shell in C. I procrastinated and wrote the whole thing in one 38-hour caffeine-fueled sprint on a 32-bit SPARC SunStation. I don't think it supported pipes but it did support input/output redirection and ANDing of commands. Not my best product, but some of my best work.

This was one of the projects in my Unix class in college. And yeah, it took about 35 minutes. :)

Reply on news.ycombinator.com