Explainshell – match command-line arguments to their help text | Hacker News Reader

Explainshell – match command-line arguments to their help text

explainshell.com

275 pointsduck82 comments

What a great idea. One of those clever ideas that are immediately obvious once you see them (but not before).

I find man pages a chore to read, and that has definitely held back my skills, so if this actually works—i.e., you can type in any command and get a readable layout of what it does—I can definitely see using it.

Edit: It might be tricky to make this robust enough across all valid commands. I tried a few variations and they didn't seem to work as well. Beware the cherry-picked example.

Also, everyone asking for pipe support is dead on. It's how most commands actually get used, so it's really needed. Plus it would take this tool to a whole new level of coolness.

Edit 2: Who made this? I think if you wanted, you could turn it into a new kind of learning environment for the shell. It already is that, in embryo, but there are a thousand directions it could go. The trouble with learning the command line is the overwhelming mass of information with no way of navigating it or distinguishing what's important from arcane detail. What's badly needed is a simple organizing principle that feels good to work with and lets you learn what you care about right now (and hides the rest). You've got that here.

An it-just-works interactive tool is a really promising way to go. There are countless tutorial articles and books, but they're one-size-fits-all, and once they get too complicated they suffer from the same problems that man pages do. A tool that can actually be used to play with and learn interactively would be a real contribution. But it might be a lot of work to turn this into that. I doubt that the corpus of "all standard Unix commands plus all their options" can be sucked up and re-presented by any generic script; the variations are too unpredictable. There would probably have to be a lot of attention put into important special cases, and a lot of careful design to get it to it-just-worksness. And that is a must-have for beginners.

Totally agree. It's definitely a "why didn't I think of that" idea.

Looking forward to the oh-my-zsh extension that puts this in my shell. Or a chrome extension that inserts it on random code snippets.

On the other hand, I did think of this, but got stuck in the perfection trap: I tried to make a parser for manpages, and never got past that.

Yeah. Thinking of something is the easy part. The devil is in the implementation.

Thanks for reviewing it.

I definitely went into this with a 'let's start with something good and useful for as many commands as possible', so I haven't focused on accuracy for those edge cases. But from what I played with it so far, results have been pretty good.

Agree again on the pipe et al. support. It's not easy, hence why I haven't added this right out of the gate. I basically have to build a parser for bash, although I can probably get away with handling only a small subset of it that is interesting in this context, such as pipes, redirections, etc. I still have to figure out how to display multiple commands, because chunking them all in one page isn't going work visually.

I'll be happy to hear more about turning this into a learning environment. It might not work for this particular tool, but maybe it can use the foundations.

Hard to follow running 1024x768: http://i.imgur.com/rVVrZCc.png

Is it possible to use the parsing in this to generate autocomplete rules?

It's possible, but I'm not sure how robust it will be. There are a lot of heuristics being used to guess what the options are, if they expect an argument, etc. So when feeding it with a command that was typed by someone, which is most likely correct, it's fairly easy to look up each option in all the possibilities. Not to mention some commands have a very complicated set of rules (e.g. find), and the position of some options is context sensitive, which explainshell isn't (it could be, but that would require a lot of manual work).

It's interesting because I had an idea once to do the opposite: see if explainshell could use completion files from bash/zsh/fish as another source of options besides the man page.

Completion scripts from bash are far too much of a kludge to extract information from.

Most tools are written in C, and the standard way to parse options in C is with getopt(3) or getopt_long(3). I actually looked into searching the compiled binary for the getopt_long structure, but that wasn't so easy. Since these functions have information on all the possible options, it would be nice if there was a standard way to get them to spit out the option list.*

See my reply to your comment in another thread. It would be a killer app to have better autocomplete support, but I can't offer assurance that it's feasible.

* I had another idea: you could use ELF tricks (LD_PRELOAD) to load a different version of getopt/getopt_long that would spit out the specifications.

Fish shell does something like this. It's nice to start from, but there are definitely not enough on their own. Git for instance needs extra rules to complete branches and so forth.

[1] http://fishshell.com/

It's about time someone made a shell for the 90s. That was a great decade. Heyyy, Macarena!

If it is a replacement for bash, then I'm afraid I'm too attached to bash to give it up.

Author here, thanks for the feedback guys!

- adding support for shell syntax is definitely the next thing on my todo list. It's a bit tricky since Python's shlex (which I'm using to parse the command line) is fairly limited in functionality and pretty much only handles quotes. So I'll have to write a mini bash parser which isn't trivial.

- the automatic options extractor isn't perfect and requires manual corrections here and there

- add an API for command line tools

I'll be happy to answer any questions you might have.

This is great.

I would love an easily found example for find. Fucking find...

I'm a pretty fair hand with regexes, if you would like any assistance, please feel free to drop me a line.

The regex is already getting out of hand ;) https://gist.github.com/idank/6382427

At some point I wanted to convert it to a DFA because I thought it will give me more control and flexibility and be more maintainable over time. The thing to look out for is not make it too permissive so it starts catching things that aren't really options.

It's funny that a similar tool to your service can be used to build it! http://www.debuggex.com/

The occasional tool, like find, is beyond the capabilities of a DFA, because it's based on a context free grammar (e.g., it counts parentheses).

This regex is the one I used to extract options from man pages, not to parse the command line (which is simply a wrapper around Python's shlex).

I see. I started doing almost the exact same thing at one time. My goal was to create autocomplete rules. I think I used the output from the tool's --help or -? rather than manpages, and I would have been using awk and sed, since I never learned Python.

Bash autocomplete specifications are messy, but you can sidestep a lot of the mess. You're allowed to run an arbitrary command to generate the completion list. I was envisioning some kind of standard option grammar that the autocomplete program would interpret for a given tool. The grammar would be partly generated automatically, but would likely need hand-tuning.

True for official regular expressions, but not for the ones used in most programming languages. http://nikic.github.io/2012/06/15/The-true-power-of-regular-...

True. Perl-style regexes have grown capabilities far beyond regexes in formal language theory. Note that I was referring to DFAs.

I wrote a very sophisticated iptables command line argument parser a while back and found shlex to be too limited. Instead I used Pythons OptionParser module--in reverse! It worked out great:

http://sourceforge.net/apps/trac/pyci/browser/iptables.py

The mechanism used by that code may or may not be helpful for you but feel free to email me (daniel.mcdougall at liftoff software dot com) if you want to collaborate. I'd love to put together a module that anyone can import to interpret/explain commands that can perform man page lookups and whatnot.

Are you aware of [http://www.shellcheck.net/]? Its pretty slick, and you may be able to work wit tat a little bit.

Code is now on github: https://github.com/idank/explainshell

For those like me who want to call it from their zsh, add the following to your .zshrc:

explain(){ curl -s $(echo "http://explainshell.com/explain/$1?args=${@:2}" | sed -e 's/ /+/g') | sed -n '/<pre/,/<\/pre>/p' | sed -s 's/<[^>]>//g' | sed -e 's/^ //g;s/ *$//g' | grep '.' | cat }

and call it, for example like: explain git status -s

Ha, cute! I'll add an API soon enough to return some JSON, so command line tools could wrap it more conveniently.

Perhaps I'm missing something, but this doesn't work for me:

% explain git status -s

explain:1: no such file or directory: http://explainshell.com/explain/git?args=status

curl: no URL specified!

sed: -e expression #1, char 11: unterminated `s' command

It's probably hn's formatting. I just tried copy/pasting from my post and I get the same error.

Try copying it from there: https://gist.github.com/ekyo/6383547

I can't edit my previous post anymore, if someone can maybe add something like "EDIT: copy/paste friendly version https://gist.github.com/ekyo/6383547" or better, fix the formatting so it works...

... I want this integrated into the shell on my ubuntu machine. Give me a big freaking sidebar panel with this thing on it.

Agreed something like

    explain tar xzvf archive.tar.gz

would be great.

Author: Do you have an API?

Once the source is released, it should be possible to make such a client without needing to connect to a web service. These breakdowns are figured out by the Python code written by the author.

That's of course great but as a child of our time I want my explain command now ;)

Posted it for zsh, should be easy to make it work for other shells:

https://news.ycombinator.com/item?id=6297957

You should be able to build this as a bash function I think.

How about this. Works on zsh and bash (at least for basic commands).

https://gist.github.com/axelson/6384869

After seeing this I've already begun writing a plugin for Gate One to do the same thing:

https://github.com/liftoff/GateOne

Soon (I hope) you'll be able to "sudo pip install gateone" on your Ubuntu machine and be able to access it in your browser on localhost (or remotely if you so desire).

I had plans to add a feature like this to the help plugin but it didn't work out and I forgot all about it. This article has suddenly renewed my interest :)

Very nice!

p.s. it looks like pipe is not supported:

http://explainshell.com/explain/cat?args=text.txt+%7C+grep+-...

Nice. Positional arg descriptions are probably hard to automatically find - e.g. http://explainshell.com/explain/tr?args=-sd+G-Z

Yes they are, and most of the time require manual tagging. I fixed 'tr', try again ;).

Looks awesome. Is it possible to list explanations in the same order as they are in the command line? For example, options for "tar xzv" are listed in the "v, z, x" reverse order.

The order of help blocks is determined by their position in the command and if they're on the left or right side of the lines.

If you always keep the order, lines would have to cross each other which looks less pretty and pretty confusing.

When I resize my browser the display order switches back and forth from xzv to vzx. Their lines connect on the left side.

In the original order the lines cross.

The f-argument is always on the bottom with its line on the right.

There is no substitute for reading source code.

grep -A20 -ri "usage:" src-directory

should get you started. But if you rely on looking only at usage(), you'll miss "undocumented" features. Try reading the OpenSSH source for example. There is at least one undocumented option.

If the source code is so voluminous and the organization of the project so convoluted that it is prohibitive to scanning through the source to figure out what the program does, then that in itself tells you something.

The smaller the program (and its source code), the easier it is to master it, in my humblest opinion.

Generally, the larger the program, the higher the probability of bugs and vulnerabilities. This, you might say, is the cost of adding "features".

I prefer small programs where I can read the source code quickly and easily. They need not have many features. The less output the better. As long as they do one thing well enough, and reliably.

Not to mention what it does to ease of use every time you add more "features" to a program. Look at the OpenSSL binary. It is like they are going for the World Record on the number of options you can cram into a single command line program. Insane.

Any reason why this is stuck in browser? I've got most oif the raw data (man pages) already on my machine and the closer this is tied to command line the more likely I am to use it. For instance I'd love to be able to do;

  explain !!

in my shell to work out what just happened!;)

Good work by the way.

This should work:

    #!/bin/bash
    cmd="$(cut -d ' ' -f 1 <<< "$@" )";
    args="$(cut -d ' ' -f 2- <<< "$@" )";
    url="http://explainshell.com/explain/$cmd?args=$args"
    explanation="$(curl -s $url | grep '<pre' | sed -E 's/<\/?[a-z]+(\ [a-z]+=\"[a-z0-9]+\")*>//g' | sed -E 's/^\ +//g')"
    echo "$explanation"

Example:

    ~$ tar xzvf archive.tar.gz
    ~$ explain !!
    explain tar xzvf archive.tar.gz
    The GNU version of the tar archiving utility
    -x, --extract, --get
    -z, --gzip, --gunzip --ungzip
    -v, --verbose
    -f, --file ARCHIVE

Save in /usr/local/bin. Gist: https://gist.github.com/backus/6387485

Beautiful. Thank you.

Absolutely fantastic. That's going to be SO helpful.

I really like what you've done with the colour coding and the lines to link up the boxes of explanation with each argument, but it is still a little difficult to see what's going on when you've got lots of args. With an example long command line [1] (the sort this is really useful for) I kept having to scroll up and down from the whole command line to the explanation.

Perhaps when you do the hover over, rather than just greying out the other options, you could hide all of them and only show the relevant one. It would be right at the top, on the same screen as the argument (assuming you had the res). You'd get explanation of the particular arg, along with it's position and context.

[1] rsync -vzrc -e ssh --exclude .svn --exclude *~ code/ target:/root/application

Loved it.

Just a tip: when the command got a long man page you can't really see the line matching the option in the top of the page and the man section down on screen.

If you guys could keep the command on the top of the screen even when scrolled down that would be superb.

I actually experimented with that but getting the lines to display correctly when scrolling was tricky.

This is amazing! Reminds me of http://www.regexper.com/.

I hope you made it maintainable, there's going to be a lot of feature requests coming your way :)

Awesome, thanks for that.

I almost never use my browser full screen, often it's snapped with 50% width. In opera chrome and firefox, the width is a tad too big for that. It'd be nice if it was more fluid.

Well that improvement wouldn't be that useful for me tough, I'm going to integrate your website in zsh and emacs if someone doesn't do it before me.

Also, if you can make it so that the input area is still there when showing results, it saves a frequent 'back' in the case someone wants to check multiple commands.

Will do, thanks!

This is really awesome, but what would be awesome as well would be the inverse, which is the ability to find potentially relevant options from man pages based on keywords.

This site is a great idea for when you want to find out what some command line black magic is, but the inverse would be great for "I know this command line tool can do this thing, but I don't know the options".

Hey, this is great! I've got a few ideas for improvements:

1. Fix the layout. At least in Safari, you can only see the beginning and ends of the colored lines.

2. Pipe support (as mentioned by Ecio78)

3. Additional Unix flavors. Red Hat and Mac OS X would probably be the top suggestions.

I don't have a Mac but I hope once the code is open someone will jump on it.

Re other flavors: do you know if there are easy ways to get a dump of all man pages, similar to Ubuntu's man page archive?

It would probably be easiest to extract them directly by running the OS in a VM. But I suppose crawling them is your next best option.

Fedora: http://linuxmanpages.net/manpages/

OS X: https://developer.apple.com/library/mac/documentation/Darwin...

Simple script to call it from command line. Uses bash, works from other shells as long as its installed. Require curl to be installed.

https://gist.github.com/ekyo/6389069

How well does it handle differences between various shells and their common tools?

Since it's called "explain /shell/", I just had to try inputting some basic powershell as a test - Sadly it's not supported yet :P

At least for PowerShell such a thing should be trivial as the shell knows about its commands and parameter parsing is consistent for every command.

I'll probably try hacking something together on the weekend for PS.

Awesome, thank you! :D

Not progressing as fast as I'd like. Stumbled over a few roadblocks regarding parameter binding. The rules are complex (and underdocumented, I might add. E.g. why man gci displays three parameter sets, or why -fi always binds to -filter and not to -file is unknown to me. CmdletInfo.ResolveParameter sounds like being great in theory but is unusable in practice. At least when I'm done I might be able to revisit parameter binding in Pash. Preliminary repo is at https://github.com/ygra/ExplainPS – once parameter binding is done the rest is just fetching information from Get-Help which is the actual trivial part.

It appears to be incorrect with oddities like ps aux, which isn't the same as ps -a -u -x since -u expects a user list (and then reads in -x as a user). But this is a weird case because of backward compatibility.

A nice-to-have would be to have explanations for non-option arguments: for example, for "sed -e s/foo/bar/", give "replace foo with bar", and even feed foo into a regex explainer.

The fish shell does some (if not all) of this interactively at the terminal.

What's the name of this functionality in fish? Might be worthwhile to install it just for that.

Not sure, but you just hit tab, i.e.:

    echo -<tab>
    --help        (Display help and exit)  -E  (Do not use backslash escaped characters)  -n  (No newline)  
    --version  (Display version and exit)  -e         (Use backslash escaped characters)

Nice. It's a shame it can't break down the more complex ssh/scp parameters such as -L [bind_address:]port:host:hostport Those are the commands I'm always Googling...

Oh wow, this is great. Many thanks! I intend to use it :) I see you already have most of my requests listed, so I'll just leave it at that and say "awesome work".

Such a very useful service, thanks! :)

It would be nice if it could detect inner commands and do a recursive parsing. Example command:

find . -name ".txt" -exec iconv -f ISO-8859-1 -t UTF-8 {} \;

Cool I also like it!

A couple of things that it should also support (or simply ignore):

- shell variables

- shell constructs (for, while, | & > ...)

- arguments followed by values without space (e.g. gcc -O3 -lmath ...)

An argument to an option without a space does work. When they don't it's because the option was incorrectly analyzed as having no arg.

Which open source license would you recommend I use?

The GNU GPL.

It allows you to share your code with other people who share their code, but not people who don't. Afterall, why would you want to share your work with someone who doesn't share theirs?

Great work, thouGh. Seriously, nice job!

this is super cool.

curious where to submit bug reports? for the heck of it I put in a long ffmpeg command; it detected the "-i" argument twice and the lines look a bit crazy.

here's the line to reproduce the error: ffmpeg -i infile -s 640x480 -vcodec h264 -r 30 -b 450k -profile:v baseline -tune film -bufsize 2000k -maxrate 550k -an -t 16.0 -passlogfile infile.mp4.passlog -pass 1 -y outfile.mp4

Hi,

I'll open source the code sometime in the future so github would be the place for bug reports.

As for long commands, yeah they're a bit crazy right now since they get wrapped. I'm still trying to figure out the best way to display it.

This is freaking amazing for educational purposes. I wish I had this years ago in software engineering.

Will show this tool to my students in the Shell course. Thanks!

Wow! This is a genuinely new and very useful idea! Congrats

Useful, thanks!

I will send this to a friend who needs to learn the basics.

Great idea, nicely done. Went to my bookmarks!

brilliant!

awesome! very good job!

Reply on news.ycombinator.com