Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool (2016) | Hacker News Reader

Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool (2016)

blog.robertelder.org

104 pointsdvfjsdhgfv32 comments

I use Perl for a lot of my scripts, so the Expect Perl module is great for automation without having to switch over to TCL.

https://metacpan.org/pod/distribution/Expect/Expect.pod

The regex matching features of the Perl version make it extremely useful to match and extract interactive results.

I've recently become a fan of expect-lite[1], which has a much simpler syntax.

Here are some examples:

  Check the IP address of the localhost

  >ifconfig
  <192\.168\.1\.1
  >ip -6 addr show
  <2001:db8::f00d

  ssh to a remote host

  >ssh root@host-021
  <assword:
  >>secret_password
  # issue a command once logged in
  >ls
  >exit

[1] - http://expect-lite.sourceforge.net/

Thank you for this, looks amazing to create simple tests

Inputting an SSH password using an expect script seems like an anti-pattern to me. In general, why can't you use a password-less key on the client and restrict what commands it can execute in the authorized_keys file on the server?

... because you don't control the server?

Expect is a great idea. Shout-out for the python version https://github.com/pexpect/pexpect

`pexpect` is used by Jupyter Notebook, and gets some maintainence from the Jupyter people. So, like the article argues, it is likely to be stable for the long term.

Years ago I was automating control of 3rd party WiFi devices, many of which presented a console which would be a much more reliable means of controlling the device than trying to hack the web UIs of the time.

I was using Tcl mostly running on Windows, and the Expect library for some reason had tremendous lag, or maybe it was a paid extension from ActiveState, I can’t remember the exact reason for not using it...

So I decided to just reimplement my own ‘expect’ command using native Tcl and a socket with just enough knowledge of Telnet control codes to get a prompt to appear.

What amazed me was how few lines of code it took to implement a fully functional Expect, and how useful it was as a means of structuring the automation, specifically around timing when to send inputs, parsing outputs, and confirming an action completed.

I’ll try to dig up the old code and post it on Github...

I used expect a ton a decade+ ago to script a bunch of ncurses-esque menus over a telnet connection. It was an obscure tool even then.

I can’t think of much I’d use it for today. Ansible is far better.

It was mostly helpful when you had to deal with a mainframe terminal program that was otherwise impossible to automate.

Also surprised to see this at HN. The approach to automating interactive applications based on pty superpose (and I/O direction) coupled to a pattern->action model is an extremely old fashioned approach. 20 years ago expect was an invaluable tool though. The model for developing cli tools has changed so much in reaction that very few use cases for 'vanilla' expect remain. Not to mention that TCL is not a very popular PL and unlikely to make a comeback.

'expect' is definitely for edge-case automation, but in the few cases where it is useful, it is extremely useful. Despite the fact that it's quite 'old fashioned', it is a dependency of DejaGnu which, itself, is a dependency of the test cases for a huge number of GNU projects. This leads me to believe that it'll still be around for a very long time and will continue to work the same as it does now.

I used it as recently as a couple years ago to make short work of some build + deployment stuff on some slightly unusual platform. Might've been Garmin wearables? Something like that.

When you need "expect", it's wonderful.

Sure, but is it worth learning TCL for a few edge-cases? I like python, so I used "pexpect" in the past. I am sure other languages have such libraries too.

And if you are going for solid production, you probably want to re-implement "expect" anyway -- so all input is whitelisted and unexpected messages are flagged. You probably don't what to break your expensive 1990's industrial device because you were driving it with expect script, and it cheerfully ignored "WARNING RESERVE BATTERY DEAD, REPLACE BEFORE POWER OFF" messages.

If all software was designed for scale, there would be no use for expect. It's 2019 and I still have expect in production:

  * to deploy commercial software whose configuration is only through  an interactive terminal session and stored in an undocumented binary format
  * to drive gdb to inspect a suspect Red Hat 6 "will not fix" GUI component that can go into an "impossible" pure cpu loop, to determine whether to kill it or whether it's busy in a more benign state.

Edit: the deploy is invoked by ansible, whose own expect module recommends the real expect "for complex cases"

fish shell uses expect as part of its interactive test suite. It's for more than just automating a workflow!

gdb has a batch processing mode and its own scripting language with conditionals, loops, etc., so you could do this entirely with gdb scripting, probably. I've had good luck using gdb scripting for various things, notably conditional-ish breakpoints that printed some debugging info only under certain complicated conditions. (I do concede that if you know expect and not gdb scripting, except is quite likely easier, and it's possible expect is still easier if you know both, depending on the problem.)

gdb can also be scripted using Python. https://sourceware.org/gdb/onlinedocs/gdb/Python.html

Edit: and Guile

For those on mobile or on MacOS w/ Safari, the two bullets above are:

- to deploy commercial software whose configuration is only through an interactive terminal session and stored in an undocumented binary format

- to drive gdb to inspect a suspect Red Hat 6 "will not fix" GUI component that can go into an "impossible" pure cpu loop, to determine whether to kill it or whether it's busy in a more benign state.

I have used it to pass passwords to ssh

I recommend using Python's "paramiko" library for this -- it has this very neat feature to allow multiple commands. You can upload the script, then run it all in one connection.

The reason that "expect" is underappreciated is right there in the text:

> most of the things that you could use expect for, are usually done much better using another method.

Outside of it's very narrow use case (driving un-scriptable CLIs and automated tests for command-line tools, and maybe some exotic CLI-centered workflows), trying to use "expect" will often result in fragile code with surprising failure mode.

In 2nd example, script will fail if user has non-default top's config. It also messes up terminal on exit. Use top's "personal configuration file" instead.

In the 4th example, you are not just trapping Ctrl-D in bash, you are trapping it in all bash-spawned commands (like "cat >file"). I am going to guess that bash's native IGNOREEOF option was what the author wanted.

In the 9th example (md5sum) , expect'll work fine -- but learning standard unix commands like "grep -m1" and/or "tee /dev/stderr" will allow many things that expect cannot, like doing an md5sum on millions of small files.

Fun fact: Don is the son of microcomputing pioneer Sol Libes:

https://www.google.com/search?q=sol+libes

Thanks. I saw the name and immediately suspect that was so.

Good times.

Don Libes is also the author of "Obfuscated C Code and Other Mysteries", which is the first book I ever read that really explores all the dark corners of programming in C; it's a really great book if you use the C language.

I used Expectk extensively in 1993, but I can't say I really enjoyed it. It was a crufty and error-prone undertaking, and in my opinion is best for testing, or as a last resort for talking to black-box code with an interactive cli.

I once used expect to brute force a puzzle in an interactive fiction game.

While running scripted `apt-get -y install ...` commands, I occasionally encounter packages that require answers to configuration questions. In some cases, the "-y" option handles the problem, but in others it does not.

I've thought about going back to the developers of the packages, asking them to provide workarounds based on environment variables, etc. I may still do this, but it seems like Expect might provide a handy workaround in the meanwhile.

However, I'm concerned that the mock-GUI dialogs that these packages use (using terminal graphics) might make it hard for Expect to interact with them. For example, some of these dialogs want the user to navigate through a menu, select an item, etc.

Suggestions, anyone?

I believe those are using whiptail, so maybe a modified version of it that allows scripting would do the trick.

expect is used to run a bunch of interactive tests for the fish shell (https://github.com/fish-shell/fish-shell/tree/master/tests, all of the files ending in .expect, where the file interactive.expect.rc defines a bunch of utilities).

Some things that would be nice to have: support for mouse events, OCR of graphical applications (so e.g. you can program it to click on a text even if the location/font of the text isn't fixed), and support for DOM tree navigation in case you're interacting with a web page.

Discussed at the time: https://news.ycombinator.com/item?id=13130218

+1. expect is awesome and so useful.

I am curious, what are you using it for?

Reply on news.ycombinator.com