ShutIt – A Python-based shell automation framework | Hacker News Reader

ShutIt – A Python-based shell automation framework

learnxinyminutes.com

145 pointszwischenzug25 comments

Open articleView on HN

Source is here:

https://github.com/ianmiell/shutit

More background here:

https://ianmiell.github.io/shutit/

This is used, for example, to automate kubernetes cluster tests for Chef scripts for OpenShift:

Chef scripts:

https://github.com/IshentRas/cookbook-openshift3

https://github.com/ianmiell/shutit-openshift-cluster

https://zwischenzugs.wordpress.com/2017/03/04/a-complete-ope...

Oh, and if you have anything you want automated, let me know - I love an automation challenge :)

Contact info please

@ianmiell

Seems like another version of `Fabric`, though the expect functionality is nice. After investing so much in learning the quirks and specifics of the fab command, I'd need a very compelling reason to switch.

There are at least some cases where the expect functionality is either required, or makes things much easier.

I know expect, or expect-like functionality is used often for automating things across groups of Cisco devices, for example. Where you need to process the output of one command to drive another, or where commands have a separate "commit" Y/N type prompt.

Outside of that, most of what this describes feels like something I would do with Ansible.

Yeah, that was an original driver (installation of a BigIron DB (Informix) on the CL).

Sounds great for pipelining data load jobs.

Message me if you have a use case @ianmiell or on GH.

As someone researching whether to use Fabric or another option, I do not have to wrestle with the sunk cost fallacy. I'd be interested to hear how the features they share compare to each other objectively.

From reading through fabric they seem pretty similar. There are a few differences that leap out:

- ShutIt's pause points/interacts allow for easier debugging

- ShutIt allows for 'multisends' where you can automate more complex interaction with non-standard prompts (eg the telnet example). It's been used to automate network tasks, and complex installation tasks.

- Backgrounding tasks and managing those for faster deployment

- ShutIt has behind it a packaging system that can leverage other scripts (advanced).

I'm not a fabric expert so happy to be corrected on these.

Or Ansible, which (as I understand) started merely as ssh automation.

There's something to be said for simpler tools in the same space though.

Suggestion to the author: try to use portalocker [1] so that the module can also work in Windows. Currently it doesn't because the code uses the fcntl module that is *nix-only. [1] https://pypi.python.org/pypi/portalocker

Thanks - did you try it in windows? I use CygWin at work, but it would be great to use it natively also.

Yes, it failed to import because of fcntl. Since you can't just replace fcntl with an equivalent win32api call, I suggest using portalocker. I usually log on *nix servers from a Windows desktop, so ShutIt could be an useful addition to my toolbox - once it works on Windows, that is.

This looks like a provisioning framework rather than 'shell automation' (scripting). I couldn't work out out if it supports pipelines.

In all honesty, who is this aimed at? Some who know python and doesn't want to switch to bash, but is happy to learn the command-line tools you use in bash?

> ShutIt was built originally to facilitate the deployment of complex Docker containers, so that developers can quickly prototype builds in a structured and flexible way with as shallow a learning curve as possible.

It's for setting up developer environments? Like a sandbox? Like docker-compose?

It's a nice API around pexpect [1] and therefor fills the same niche - spawning and interacting with console applications in a programmable way.

[1]: https://pexpect.readthedocs.io/en/stable/

I'm familiar with expect, but I don't have any modems anymore which require AT commands to be sent to it with precise timing.

It seems like you are describing what it is, not what it should be used for (which is what I can't figure out, based on my knowledge of other tools).

expect was used for a lot more than modem interactions. Certainly, there are many ways to skin this cat today, but it's not crazy to want something that removes the boilerplate of capturing output from commands and figuring out how to proceed to the next step based on what it gets.

To answer your question with specifics, you could use something like this to:

- Start services, and then do something (maybe alert, maybe make a change and try again) if it failed.

- Make configuration changes across several files and check the results of the changes (i.e. did things continue to work after the change).

- Check for updates in version control and then optionally deploy them based on the results (e.g. if there's a new tagged version).

- Interact with some API periodically, and do something if results change.

- Tons of things I don't do, but somebody else might. It's kind of an IFTTT (If This Then That) for the command line (which is maybe obvious, since I guess IFTTT was an implementation of expect for the web, in the long tradition of "take a UNIX utility and make a version for the web"). It isn't for any one thing; it's for whatever thing you want to programmatically do with stuff that maybe wasn't meant to be programmatically worked with (or maybe just doesn't have a Python API).

All of this can be done with any reasonably competent programming language, including Python (I use Perl with some tiny custom libs or bash with boilerplate, usually). This removes a bit of code. Maybe only a little code is removed, maybe a lot. It depends on what you're doing, and how well the commands or APIs you're calling lend themselves to being called from a shell or Python or whatever.

Expect was a tool to make working with tools that maybe didn't intend to be called from shell (or Tcl, or C, for that matter) programs work better when called from programs. There have always been many ways to do it. If this doesn't fit the way you think or the tools you use don't need it, there may be better ways for you to accomplish similar tasks. But, it's not a crazy idea.

Thanks, and well put.

Here's some things I've done (with the entire framework also - the OP is the simplified 'standalone' version):

- Developed the 'ShutItFile', eg: https://zwischenzugs.wordpress.com/2017/03/18/clustered-vm-t...