Expect – Linux tool for automating interactive programs | Hacker News Reader

Expect – Linux tool for automating interactive programs

linux.die.net

171 pointsactinium22687 comments

Expect is such a great tool for a specific class of problem. Decades ago I worked as a test engineer for a big hardware vendor. We had maybe a couple hundred raid controllers that we were testing and to address some issues we found, the supplier needed us to change a bunch of configuration variables that were only accessible via a text menu system over a serial connection to these devices. After painful manual reconfiguration of a handful of these things, I instead wrote a fairly complex expect script to make the appropriate changes and also validate all the settings on the rest of the raid controllers in a fraction of the time it would have taken to do it manually. It saved me so much hassle and got me some pretty good kudos and respect from my direct leads. It's one of those tools that I kind of wish I would need to use again!

this is the sort of context I've also seen expect use. hardware test automation in manufacturing xD

One of the early papers by Don Libes, author of Expect, explaining how to use it; published 1990 - thus predating Linux: https://www.nist.gov/publications/using-expect-automate-syst...

Linux tool? Hate when everything Unix automatically gets categorized as Linux. Have used Expect for many years for many use cases from automating servers to networking use cases as well.

I’ve used Expect on Windows systems. Active State provided commercial distributions of TCL for Windows.

I used Tcl and Expect on Solaris at a job in 1995

I used to use it on HP-UX in the 90s.

Poor soul.

<https://news.ycombinator.com/item?id=41272562>

For example, expect(1) is described in the 1989 Don Libes book "Life with UNIX". Libes is the author of Expect. A title such as "Life with Linux" would probably have made little sense at the time.

Correction: I went back and looked at the book and it does not mention expect. According to Wikipedia expect was first released the following year, in 1990. Apologies for the inadvertence.

Nevethless I think it is safe to say that expect was not written using or for Linux as Linux was not released until 1991.

> Hate when everything Unix automatically gets categorized as Linu

Those days, people are thought to not bother their brain with useless words and knowledge.

My go-to example for `expect` is password-based SSH. Since we're all enlightened beings these days who surely never use this anymore, there's no risk to it:

    #!/bin/bash

    expect -c "
    spawn ssh user@10.0.0.1

    expect \"password:\"
    send \"12345\r\"

    interact
    "

`interact` hands you back control to do whatever you want, but you don't actually need to do that. You can have `expect` run every command, fill out every field.

The tool you need when you're in environments you don't deserve.

For this particular use case, I'd suggest sshpass instead.

  https://linux.die.net/man/1/sshpass

...

Operations folks today might not understand -- everything just works nowadays, which is slightly boring. And boring is good!

But there's something unique about the feeling of making things work even though they are not designed to work, and really have no right to work. Expect has been a lifesaver for some of the most critical systems at some of the biggest companies in the world.

No aspersions cast by me!

Definitely, `sshpass` is the more better keyhole tool for this job. I picked it as the most common example I can imagine everyone going "ugh, I have to do this by hand again?" at.

sshpass is a travesty of security.

It has all the pitfalls of putting your front door key under your doormat. Except you're doing it in a world where everyone can materialize a unique key out of thin air, and you can always instantly tell your door which of those keys should or shouldn't be allowed to open it.

sshpass is a curse, even calling it a crutch would be improper flattery. It's a dangerous cheat that accomplishes nothing except impeding people from otherwise spending the 20 minutes it takes to figure out SSH keys.

sometimes the device you're connecting to doesnt support ssh keys.

Yes this is the proper dogma, but you're missing the point.

Any automation around passwords is a crutch and a mistake. But sometimes it is necessary.

You don't always control the remote systems. The remote systems are not always capable of key-based auth. And sometimes the remote system is not of high concern so the "danger" is null.

sshpass makes a reasonable effort to do the best-possible thing under these less-than-ideal circumstances. The other options suck more.

My most recent use of sshpass is to collect reports from a vendor over sftp. I would have preferred to use https with BASIC auth, but in truth that has exactly the same problems as sshpass, and I have other hills to die on.

You forgot all the other prompts that ssh might produce before the password prompt (e.g. add new host to known hosts yes/no). There are more possible prompts than most people realize.

Darn. If only there were a tool that can interactively answer prompts...

This isn't a script that we'd send on an unmanned, inter-planetary, mission. This could be something that can save the tedium of typing password on a daily basis. If it fails, well, we still have a reason to be alive: press ^C.

You can skip the new host prompt with ssh -o options.

For the case of entering passwords, you can just use ssh authentication instead

There's certain devices (e.g. firewalls) that don't allow you to save your public key for SSH authentication, so you're stuck with passwords. I've used "expect" for automating backups to a TFTP server.

out of curiosity, why do you call the expect script in string form from a bash shebang, rather than make the script itself an expect script directly with an explicit expect shebang?

I have no technical reason, I just wanted to wrap it into a familiar context for people. People are I find less scared of "useful CLI tool I can script around" than "scripting language in its own right".

I used expect a ton years ago, for AIX system janitoring. A lot of things required telnet’ing into proto-ncurses mainframe applications and going through menus, typing and selecting stuff etc.

The actual productive workflow is to do your task with autoexpect, and then refine the generated output: https://linux.die.net/man/1/autoexpect

I really rail to understand why modern unix OS scripting has not made expect obsolete. But then again bash is state of the art, right?

Meta shells and meta meta shells should be an assumed feature by now, not a hit or miss of terminal program feature sets, and not tied to UI programs.

Linux – UNIX-like tool for running expect and other portable software

;)

Many years ago I used the Perl version of Expect (https://metacpan.org/pod/Expect) to automate several workflows that wanted manual inputs.

I thought that it was far better than the TCL Expect because I could also use other CPAN modules and have Perl's regular expressions to parse the system's output.

Expect is a great tool to have in your toolkit.

> Perl version of Expect (https://metacpan.org/pod/Expect)

Still a great answer and latest update in March this year.

Arguably, Tcl can do GUIs much easily and can integrate C less awkwardly than Perl. Both have their use cases, at any rate, and more people should use Tcl, Perl and any version of Expect. Nowadays it falls into the category of tools that were once somewhat used, but now are obscure, like fold/fmt, nl, comm and pr.

>> Arguably, Tcl can do GUIs much easily and can integrate C less awkwardly than Perl.

We used Perl/Tk for GUIs on that old system. It brimgs the ease of Tk GUIs to Perl:

https://metacpan.org/dist/Tk/view/pod/UserGuide.pod

>> Nowadays it falls into the category of tools that were once somewhat used, but now are obscure

To be sure, but a lot of it still works without changes!

Perl/Tk has always been a red headed stepchild since it is a fork attempting to implement a C API that doesn't require Tcl to be linked. Because of that, it has long missed out on significant improvements to Tk over the years. Python Tkinter avoids that by just wrapping around a Tcl interpreter.

> Perl's regular expressions to parse […]

To be fair, Tcl has Henry Spencer’s regexp[0][1], and is awesome; Postgres also base their regex on Henry’s work for Tcl[2]. Perl is certainly famous for its regex and popularizing thinking of problems in terms of regexes (for better or for worse[3]), but it’s not the only game in town.

[0] https://en.wikipedia.org/wiki/Henry_Spencer

[1] https://wiki.tcl-lang.org/page/Regular+Expressions

[2] https://github.com/postgres/postgres/blob/master/src/backend...

[3] https://stackoverflow.com/questions/1732348/regex-match-open...

>> Perl is certainly famous for its regex and popularizing thinking of problems in terms of regexes (for better or for worse), but it’s not the only game in town.

Agreed.

In that old system we did not have any TCL scripts or third-party libraries--just C, Perl, and C shell (no Bash; it was Unix).

C shell scripting is painful (https://www.grymoire.com/Unix/Csh.html), so practically all scripting was Perl or occasionally AWK.

I use pexpect frequently for automating serial-port communications. It’s very useful for implementing test/flashing automation that needs some kind of a serial-port step (such as asking u-boot to run a few commands).

I do something very similar, how do you deal with random serial port data loss (dropped bytes)? Maybe it is specific to my environment, but I get mangled commands about 1% of time, which really sucks for automated workflows. I mostly countered this by writing commands to a tmp file, verifying md5sum and then executing the file on the device side.

Serial is slow but typically very reliable. If you’re seeing random dropped bytes, I’d take a look at your wiring and PC interface UART which might be low quality. FTDI UARTs in particular are often counterfeit (the real ones are good).

Across a large enough sample set, I definitely see occasional serial errors (although they're rare).

For me, I detect them by checking the exit-code on each command that I run, and also by looking for expected output in response to the commands.

Corrective action is usually a full-stage retry.

expect is one of those true love/hate things. Probably because the situations in which you need to use it are likely cursed. My time spent dealing with expect scripts is always miserable, and yet I don't know what I'd do without it

I've mostly used it for testing and scripting CLIs. Are there any better ways to do that?

try using charm bracelets vhs tool to script the tests, then get the demo video as an animated gif for free.

> the situations in which you need to use it are likely cursed.

So true, and yet so common.

Expect predates Linux by years. It is an old school portable program.

It looks like Expect was first released in 1990 and Linux in 1991 although Linux wasn't really that usable until around 1993.

I don't think it's unfair to say that they iterated on Linux until it could run expect :-)

Yeah, I remember reading some posts around 1993 where someone wanted to port some existing Unix program to Linux. Instead of modifying the program they added the missing functionality to Linux. I think it was about polling and signal handling or something like that.

I just used python pexpect [0] for a fun project. I believe pexpect was inspired by the original Unix expect command.

I used it to let aider suggest and run shell commands [1]. This is handy, so aider can offer to run the code it just wrote for you, run migrations, install dependencies, etc.

Using pexepect makes it easy for aider to run interactive shell commands and capture their output. This allows aider to fix bugs or suggest next steps based on the command’s output, error messages, etc.

[0] https://pexpect.readthedocs.io/en/stable/

[1] https://x.com/paulgauthier/status/1827012590406607040

wait so if i understand you correctly,

1. aider makes some change 2. aider then tells itself to use pexpect to check what changes it made 3. pexpect runs, opens browser somehow captures error messages if any and feeds that back?

i am pretty sure i misunderstood something

There’s a little demo video in the Twitter link that might be worth checking out.

But ya, aider can: code something for you, run it, capture the terminal output with pexpect and iterate further based on that.

The only problem we had with pexpect (maybe expect) was that we started to have prompts disappear or text not appear/be presented to our python script. Could have been an issue of buffers not flushing. It’s been several years but this one detail I remember because we had several workarounds for it.

https://github.com/rwmjones/miniexpect Miniexpect, a tiny C library that does something similar (for programs, not scripts).

Expect used to be one of the "big" tools at one point. O'Reilly books written about it and so on. But then it went to obscurity. Kind of strange.

It's a rare thing nowadays to find a terminal command that is interactive and doesn't provide alternative flags for batch processing.

Expect would still be useful for someone scripting a debugger (for example), but Tcl falls under the close to isoteric language nowadays and it takes less time to just repeat the commands in a readline capable interactive program than find just the right way to script it.

Last time I've tried to use expect was when I attempted to build a Tk frontend to the Haskell GHCi debugger (similar to the DDD application), but I got tripped up, from vague memory, on terminal ansi codes, and random lock ups I couldn't pinpoint.

Some tools like ssh and gpg go out of their way to prevent easy use via command line flags. They often have a good reason for doing this, but there are also sometimes good reasons where this type of protection causes problems. Since the implementers won’t budge on their stance, you need workarounds like expect.

the windows gui equivalent is AutoHotkey https://www.autohotkey.com/

We used pexpect to automate deployment from various servers and it would prompt users for passwords then they could login to perform deployment tasks for our software. It powered our entire deployment arch at that time. But it would frequently drop off text or prompts if you started doing nested logins.

We rewrote our deployment arch since then

Legendary tool. I used it once to automate an integration between Sailpoint IIQ and VMware's vRealize Automation.

Dejagnu, written in expect, is also useful for testing. I have used it in my TUI personal organizer project.

https://en.wikipedia.org/wiki/Dejagnu

Tcl/Tk and Expect were popular back in the day because TCL could perform asynchronous programming most easily.

claude.ai just suggested this as a tool for automating a workflow that involves ssh'ing into another machine and running some commands as sudo. I'm surprised I've never heard of this tool before. It works great for this scenario and the script can be used a reference to go in manually.

What the hell. If the device you are logging into supports sudo then it probably supports bash or python too. In that case a better solution is to create a script on the target device and then call that script over ssh, using key-based authentication. You do not need expect. Expect is mainly used on devices with a rigid terminal UI that do not support scripting and require interactive password authentication.

That's definitely do-able, but I didn't want to do that because it adds a level of indirection. When I put my relevant commands in a script, I have to start thinking about what directory it's in and how that affects its operations. I also have to think about how to handle errors for failures of the various commands, and I can only handle those errors on the target machine. If I'd doing expect and a command has an error, I can handle it on my local machine which might have more info (maybe I can set it up to switch to a backup machine or something). Are these solvable problems? Sure, for me it's a matter of taste.

Also, your comment would read almost identically and perhaps even better without the first three words.

Expect was the first tool that really hooked me at my first job at Sun. I had to write a tool to automate verification of the boot process during firmware dev, handling variety of odd/unexpected behaviors, paths, and prompts.

Many years later I heard they were still relying on that tool!

What until you learn about the command "yes" -- now you're really automating with gas!

Expect helped launch my early career in programming.

Automating tedious tasks as a network engineer and sysadmin motivated me to learn proper software development. If gluing together TCL/tk scripts could make my life easier, imagine what larger programs would do.

The good days.

Very handy tool. I use it to partially automate logins on a CLI tool that have 2FA. Expect hooks op (1Pass’ CLI) to send the password, then tells the program what method of 2FA I want to use (Okta push to my phone).

Same here - fetch the password from mac's "security" command, and then select the 2fa method.

> Cause your computer to dial you back, so that you can login without paying for the call.

I don't understand what this means, but it sounds like an interesting piece of 'how we used to do things'?

In the days of landline service, you have to pay for long distance calls; so it mattered which side made the call, as the originator of the call was charged. So you make a quick call to have the computer call you back and stay on for a long time; the number associated with the computer location is billed.

I find I am able to duplicate basic expect functionality by running tmux on the server and using tmux commands, namely tmux send and tmux capture.

Where the Tool Command Language really shines is this, commanding tools. TCL is criminally underrated for that use case.

MacPorts is a noteworthy project that seems to use Tcl in its sweet spot.

> Cause your computer to dial you back, so that you can login without paying for the call.

oh. that's an OLD old tool..

One of the greatest tool for the shell, I womder how many jobs this tool help to keep

Good example of an LLM suggesting something useful, even if just RTFM for forgotten tools.

O’Reilly put out a book about expect back in the old days (1995).

I used expect a few times to automate remote admin, and even to scrape terminal-based applications.

What? It's a manual page.

OP here, you must have missed my original comment where I said that claude.ai suggested this to me.

Is there an alternative to expect that I can just compile locally without having to deal with dependencies?

expect predates Linux!

Now add cross-platform GUI interaction for apps and web apps. [Ducks]

As a default, I'd keep everything in one language: https://pypi.org/project/pyexpect/

pexpect seems to be much more popular though

I don’t think you understand nor used the program expect before

Pyexpect is in another class of software. It is a Python testing utility that provides expressive assertions for unit testing.

On the other hand, the Unix expect program, and its Python equivalent pexpect, are used for automating interactive applications.

I suppose he/she meant pexpect https://pypi.org/project/pexpect/

Thank you

Reply on news.ycombinator.com