Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

If you have a slave device over RS232, expect is probably your only option. But if that remote item is running sshd, you are way better off without expect.

When I was a graduate Comp Sci TA in the 1990's, it was the perfect way to automate grading the Unix shell that the Systems Software class was supposed to write. Effectively, I was also writing a regression/functional testing system for the class. I explained that the Expect/Tcl script was going to be used to grade their project, and I provided the code.

One of my students said I was the best TA ever! There were a few students who complained that they shouldn't be marked off for a feature breaking, because it had worked a week or two ago. There were a few students who never figured out that the script was going to be used to grade their project, even though I provided weekly email updates explaining this.

Also:

1. I had read much of Don Libes' Expect book. He had a unique style of writing in that book. Hard to describe, partly since it was some years ago, and partly because I can't find the words for it. But I think anyone reading the book would notice the style after reading a few pages. It was not a bad style by any means, just ... something different.

2. I've read that expect was (maybe still is) used heavily in the electronics/EDA industry, though I am either not sure or don't remember how exactly they use it, or rather, for what. And also maybe it was used heavily by GNU software for testing. Maybe someone here can confirm.

In some cases you can deal with it with some "printf 'line1\nline2\n'| vendor_command", but the vendor command may ignore stdin and you have to use expect.

Having to automate that way is extremely brittle, stdout and stdin are very poor and unstable APIs. For example I remember some internationalized installers which gave stdout outputs according to your locales (lesson learned, export LC_ALL=en_US.utf8 before an expect script). And if one message is added or is changed, it can break...

Personally, I tend to repackage badly shipped software (yay installanywhere crap...) in deb or rpm files, at least that way, I encounter issues before I deploy in production and I can update/remove this software easily and completely. However, it only solves part of the problem, if the configuration steps are interactive only, you've to resort to expect in production.

Expect is less and less useful, it's a last resort solution, but when you need it's extremely helpful.

https://github.com/ianmiell/autopexpect

http://expect.sourceforge.net/example/autoexpect.man.html

Silly backronym name aside, RANCID is a very handy tool and uses expect to do a lot of the 'dirty work'.

- If there are alternatives, they'll usually be better

- It'll still be around when there are no alternatives

- Some tasks (e.g. "toggle foo mode") are inherently easier than others (e.g. "press up five times and hope that's enough")

- Some tasks need frequent maintenance and tweaking (e.g. as i/o formats change, program features are added/rearranged, shell environments are tweaked, etc.)

- Some tasks work so well that they're taken for granted for a decade, and it's only when you want to tweak some part that you remember how truly horrifying the implementation is ;)

Essentially, expect scripts run a program, send it some input, look for some regexes on stdout, and then send more input, which may change depending on what the regex matches. And so on.

This is useful, say, if you have a package management tool that requires you to explicitly confirm every update. Or if you want to automatically drive a shell on a remote machine over SSH, and you can't use ssh -c for some reason.

You don't need it often. But when you do, it's very useful.

eg. Running the installer displays a licence agreement, then prompts to accept or decline.

people have just dropped tcl in favor of perl and later python and others ... I don't personally like tcl anymore, but I programmed tcl/expect in the 90's and enjoyed it a lot. I now co-maintain pexpect.

    $ info Groff | awk '$2 == "History"{f = 4};f && f--' RS= ORS='\n\n'
    1.2 History
    ===========
    
    'troff' can trace its origins back to a formatting program called
    'RUNOFF', written by Jerry Saltzer, which ran on the CTSS (_Compatible
    Time Sharing System_, a project of MIT, the Massachusetts Institute of
    Technology) in the mid-sixties.(1)  (*note History-Footnote-1::) The
    name came from the use of the phrase "run off a document", meaning to
    print it out.  Bob Morris ported it to the 635 architecture and called
    the program 'roff' (an abbreviation of 'runoff').  It was rewritten as
    'rf' for the PDP-7 (before having UNIX), and at the same time (1969),
    Doug McIllroy rewrote an extended and simplified version of 'roff' in
    the BCPL programming language.
    
       In 1971, the UNIX developers wanted to get a PDP-11, and to justify
    the cost, proposed the development of a document formatting system for
    the AT&T patents division.  This first formatting program was a
    reimplementation of McIllroy's 'roff', written by J. F. Ossanna.

If the system has a CLI, you write a script in that CLI; you don't write an expect script.

Sometimes systems have CLI, but only behind a remote access wall (telnet, serial, ...); then you might use expect---because there is no way to upload a script and dispatch it.