Click – Python library for command-line interfaces | Hacker News Reader

Click – Python library for command-line interfaces

click.pocoo.org

370 pointsgregnavis102 comments

I did not expect this to be on hackernews this early. I want to point out that I have not made a release yet and it's not yet feature complete. Mainly I want to ask for feedback on the general design.

I have a very similar library for the Dart programming language:

https://github.com/seaneagan/unscripted

Checkout the github issues there for some features I want to add, like bash completion support for example. I think we can steal ideas from each other!

One nice thing about dart's annotations vs. python's decorators is they can be placed on a function's parameters as well, so it allows the option/flag/argument declarations to be a bit more DRY.

Armin, my feedback is in the form of my own version of this library.[1] I have been working on this on and off for some time and like you have not made a release. That said, I have been using it quite a bit both in my research and at work and I think it helps.

The main idea behind it is make it easy to write a "getopt" style program with arbitrary command nesting. I have found that although argparse and sisters are nice libraries they don't allow me to do many of things I like to do in my interfaces. For instance sometimes like options such as

    foo -x a -x y -x q ...

where I would process that into like so:

    extras = list()
    for opt, arg in opts:
        if opt in ('-h', '--help',):
            util.usage()
        ...
        elif opt in ('-x', '--extra'):
            extras.append(validate_or_die(arg))

I also believe that you should have "fast fail" validators. So I have several in `optutils` which are like:

    util.assert_dir_exists(path)

which if a directory doesn't exist on the path it creates it. If there is already a file there and it isn't a directory it dies with an error. When it dies, I try and have unique exit codes for various errors (for testability) and provide usage information immediately. This style is nice because it provides immediate feedback to the user with no fuss. I think a lot of "option parser frameworks" miss the point in having lots of things for parsing ints and things. Most of the time I deal with files, directories, and "string" parameters which these libraries don't help with.

In general, the standard libraries make it way to hard to write really nice command line tools. I like some things about your library, but I think that you need to increase the flexibitly for how options are processsed to you can do whatever you want with them. I also think that option parsing and configuration should be integrated. I am working to support that but I am not there yet. (see optutils/conf.py for my current ideas)

[1] https://github.com/timtadh/optutils

> I have found that although argparse and sisters are nice libraries they don't allow me to do many of things I like to do in my interfaces. For instance sometimes like options such as

This is trivial to do with argparse (or optparse for that matter):

    import argparse

    def a_prefixed(string):
        if not string.startswith('a'):
            raise argparse.ArgumentTypeError("%r does not start with 'a'" % string)
        return string

    parser = argparse.ArgumentParser()
    parser.add_argument('-x', '--extra', action='append', type=a_prefixed)

    print parser.parse_args()

resulting in:

    > python test.py -x afoo -x abar
    Namespace(extra=['afoo', 'abar'])
    > python test.py -x afoo -x baz
    usage: test.py [-h] [-x EXTRA]
    test.py: error: argument -x/--extra: 'baz' does not start with 'a'

> I also believe that you should have "fast fail" validators.

Isn't that what the callback parameter is for, especially with is_eager=True? Or ParamType if you need either something more reusable or something more extensive.

> Most of the time I deal with files, directories, and "string" parameters which these libraries don't help with.

http://click.pocoo.org/api/#click.File, argparse has something similar.

@masklinn good point. I haven't used argparse (mostly out of compatibility requirements with 2.6). So I may have mis-characterized the state of the art. It does do many things well. One note, is that the `prefix parsing` functionality could cause weird behavior when doing partial parsing (which I do a lot of).

> Isn't that what the callback parameter is for, especially with is_eager=True? Or ParamType if you need either something more reusable or something more extensive.

yes. I think almost everything should work like this. I think the getopt style makes this a bit easier to understand.

> http://click.pocoo.org/api/#click.File, argparse has something similar.

Not at all the same. I never said my programs were going to open the files themselves. I often have to write automation scripts around other things. In these cases I need to make sure files and directories are sane but I don't open them. I just canonicalize them and pass them on.

The big thing is the lack of integration with configuration files which is something I am still working on myself.

I think the creation of a group by decorating an empty function seems awkward. Why not just have the group be an ordinary class, i.e. something like "cli = click.group()"?

Consider adding mechanisms to facilitate bash autocomplete. :-)

Gorgeous. This guy has a full time job, a wife, and wrote Flask and Werkzeug, ItsDangerous, Sphinx, Markupsafe, Jinja2 (and Jinja) and that's just some of the well known stuff. Handsome and young too. Bastard. He probably saves the world in a dinner jacket in his spare time.

Sphinx was written by Georg Brandl, not by Armin Ronacher.

Hmmm. It's listed on his projects page...... https://lucumr.pocoo.org/projects/

I worked with Georg on it, but the bulk of the code is from him.

To be fair he only got married this month, so the jury is still out on where his productivity goes from here ;)

Getting married didn't kill me it was having a kid. Now I get maybe four hours a week for hobby/OSS programming.

This is true. I have two now. Personal time is now a dusty concept.

Given that I wrote parts of this on my honeymoon I think it helped. Fingers crossed :)

You never cease to amaze :D

That's actually not a good sign for future marital bliss.

o rly? ;)

Armin Ronacher is absolutely amazing at python. If you want to learn some things dive into his source code and dig away. He really thinks through code architecture.

I am already on this. Not gonna stop till I read up on all of his code.

Its a plane, its a bird, No its kick ass python programmer. :).

Not only that, but Armin is a master at writing documentation for his libraries. I wish I could be as good as him.

It looks really well done.

I still prefer docopt (https://github.com/docopt/docopt) . It is so much simpler to use. Click seems to be a bit overengineered.

As someone who has never used either, the documentation and simple example of Click won hands down.

I could use it for simple cases within 15 seconds of reading. Docopt not so much.

You're right - looking on the http://docopt.org/ page it's not immediately obvious how it works.

The general idea is that you don't specify any code, you just give the help message as text. Docopt parses that to figure out all the options etc and convert those into the parameters coming in to your system.

It's a really clever idea. You specify the human interface, docopt converts that into the code version (so long as you adhere to a few common conventions). I haven't seen a cleaner system anywhere.

From their example:

    """Naval Fate.
    
    Usage:
      naval_fate.py ship new <name>...
      naval_fate.py ship <name> move <x> <y> [--speed=<kn>]
      naval_fate.py ship shoot <x> <y>
      naval_fate.py mine (set|remove) <x> <y> [--moored | --drifting]
      naval_fate.py (-h | --help)
      naval_fate.py --version
    
    Options:
      -h --help     Show this screen.
      --version     Show version.
      --speed=<kn>  Speed in knots [default: 10].
      --moored      Moored (anchored) mine.
      --drifting    Drifting mine.
    
    """
    from docopt import docopt
    
    
    if __name__ == '__main__':
        arguments = docopt(__doc__, version='Naval Fate 2.0')
        print(arguments)

That's brilliant. I wonder if there's a way to use docstrings like that to create REST services in Flask.

I've used docopt for some tools, but argtools (and click it seems) is much more convenient, imo. It's very easy to remember the 1-2 decorators and probably most importantly, the command arguments and everything else are where the command is defined, not somewhere else.

When using docopt i spent much more time defining the docstring and figuring out the right way to write it, especially more complex scenarios.

I love docopt. My big draw is that I write the documentation only once - in the doc string where it should be. Click looks great, but there is no documentation in the code itself. You need to run the code to print the documentation.

If you need to parse something as simple as `foo --option spam --option eggs` remembering both spam and eggs, docopt breaks down. You are also unable to compose an interface using docopt, if you want to allow plugins in a plugin system to add their own options for example.

docopt is much simpler because it's only capable of creating very trivial interfaces. That's perfectly fine if that's all what you want to do but it also makes docopt unusable, if that's not the case.

Maybe I'm misunderstanding but you could do:

    Usage:
      script.py --option=<name>...

    script.py --option spam --option eggs

    {
      "--option": [
        "spam", 
        "eggs"
      ]
    }

Example here [0].

I haven't pushed it too far so I'm sure there are other cases where you'd need to create a different command line api to have it work (that may or may not be an issue depending on your use case).

Regarding the other criticism, seems valid. In the docopt api can you have it do the parsing for you so you can edit the rules afterwards before you run them?

[0] http://try.docopt.org/?doc=Usage%3A%0D%0A++script.py+--optio...

Does anybody know docopt? I really like it because you simply write the help/usage as text and docopt automatically generates the parser for it.

Take a look at the example in the README: https://github.com/docopt/docopt

I'm using docopt....It's really nice with it's automagicness. But for some stuff it's just not enough, too unflexible. I'm really excited about click.

It seems to me that there are a few projects similar this. Here is another, https://pypi.python.org/pypi/Cogs/ (conceptualized as a "Makefile" replacement).

I'm wondering if there could be a breakout at the next PyCon to see if we could discuss approaches and come up with a unified way to do convert Python libraries into command line scripts?

Yeah, plenty:

- Cliff http://cliff.readthedocs.org/en/latest/

- docopt http://docopt.org

- argparse

- optparse

etc etc etc...

There's also Clint by Kenneth Reitz (https://github.com/kennethreitz/clint)

Clint is a cmd tool not parsing tool

And the author acknowledges that:

"There are many alternatives to click and you can have a look at them if you enjoy them better. The obvious ones are optparse and argparse from the standard library.

click is actually implemented as a wrapper around optparse and does not implement any parsing itself. The reason it’s not based on argparse is that argparse‘s design does not allow proper nesting of commands by design and has some deficiencies when it comes to POSIX compliant argument handling."

What I love about Pocoo is they always have stellar documentation and give clear rationale - from the beginning.

Does that mean optparse will be maintained along with click? optparse has been deprecated in favor of argparse since 2.7/3.2.

Given that argparse has known problems and that not all optparse code can be ported to argparse I doubt optparse will ever go away. It's also a very small module and if optparse will ever disappear only a subset of optparse is needed to make click work.

And if you can't do it with docopt maybe you are doing it wrong.

Kudos again to the pocoo team for creating such a simple and useful library. It sure beats optparse :)

On a side note, however, does anyone know why the team prefers to wrap functions in decorators? Flask also uses them, but what's the design decision behind them?

Decorators are so simple and minimalistic and elegant and easy to understand. They can surface an arbitrary amount of extra functionality with a single statement. For example, ensure a function is only accessible to authenticated users by simply saying something like @requires_authenticated_user. Another example would be to create a logging decorator to wrap a function or a class to create a log of every time the function is accessed @log_function_access. Or to wrap a function to ensure that every access to the properties of that function are pushed through some sort of validation or transformation @validate_set or @transform_get. Decorators allow encapsulation of functions within other functional concepts with almost no code, that's the design decision for using them.

Yep, it's best us case is basically acting as pythons anonymous function (since lambdas suck for everything but the most trivial things).

Where in js flask might read app.route('/', function(){//do something}); python uses a decorator right above a regular function definition. The alternative is to manually add it after it's defined (in flask that's add_url_rule), but that kind of hides the intent of the function and is more cumbersome as you need to manually pass in the fn name.

This is a valid question. When things become complex I found that decorator is a bit too magic, and should be used with moderation.

On a related note. Which python library would you recommend for text-based interfaces? I have never used ncurses, so I don't know how complex it is. What I would like to achieve is having a user launch my script from the command line, use the text based interface to select a source and destination folder, set a few parameters and show a progress bar.

I'd say http://urwid.org/ should be the best choice for something like that. The main alternative would be to hack it together by yourself using https://docs.python.org/3.4/library/curses.html

Urwid is definitely the nicest of the curses widget frameworks I've used for python.

https://github.com/thomasballinger/curtsies looks well-designed. (I haven't tried it yet. What I do is code to ANSI terminal codes directly: e.g. https://github.com/darius/sketchbook/blob/master/misc/sokoba...)

It's also possible and very easy to do this with python-dialog[0] or with whiptail[1] (no progress bar).

downside: both need external binarys.

[0]: http://pythondialog.sourceforge.net/ [1]: https://github.com/marwano/whiptail

>You can get the library directly from PyPI:

>pip install click

Well... no you cannot. See the page: https://pypi.python.org/pypi/click

The package hasn't been uploaded yet. However one can install it straight from the git repo:

> pip install git+ssh://git@github.com:mitsuhiko/click.git

Can someone comment regarding using pip like that? Is it fine to put this on req.txt? Any best practices somewhere?

That syntax is usually meant for packages that you want to edit after installation (with -e).

If you plan to release your stuff, the dependencies in your req.txt should be as pinned as possible. The classic example is Requests: when it changed the API fairly significantly, umpteen installers broke... just because people did not bother with specifying a version for that lib.

You can also install straight from a github tarballs:

pip install "https://github.com/mitsuhiko/click/tarball/master#egg=click"

to pin a specific revision:

pip install "https://github.com/mitsuhiko/click/tarball/5b7b7296fabc5d47d...

(btw, it seems bad practice to add such a link as a dependency in your setup.py... usually you'd do it for temporary shallow forks, but otherwise I think it'd be better to also upload your shallow fork on pypi ...I guess you can just remove it from pypi if it won't be needed anymore)

I think that's because it hasn't been released yet. The author(s) may just want some feedback or opinions first.

A bit off-topic, but I like how pocoo.org uses different fonts for different projects:

Flask: Georgia for text, Garamond for titles

Werkzeug: Lucida Grande for text, Ubuntu for titles

And now click: Ubuntu Mono for text, Open Sans for titles

I had the same thought, and will almost certainly be cribbing this page's body-text font stack...

font-family: 'Ubuntu Mono','Consolas','Menlo','Deja Vu Sans Mono','Bitstream Vera Sans Mono';

...(fulfilled by Menlo on my Mac) for a future project.

Why is it that (nearly) every description I read about some random new Python command wrapper fails to get the "python" and ".py" out of the command examples, even in Linux?

I don't blame this particular offering, since I don't think release was actually planned just yet and any number of other projects have made the same subtle mistake.

Command Name Extensions are Harmful. Don't expose such an implementation detail in every example, lest everyone actually follow them. Use the "#!/usr/bin/env python" or whatever at the top of your scripts. And yes, you can keep the .py if what you have is a library, not just a command (but it's nice to then make a wrapper the doesn't expose the implementation language). And obviously in other OSes where the command extension can be omitted and still work this isn't such a big deal.

But in Unix/Linux, commands should be reimplementable in a different language without making some .(extension) a like, retained to keep from breaking other things that depend on it. Just say no :-)

Did you see the last part on setuptools?

It actually shows you how to set it up so you don't even use a #!, but rely on setuptools to make an executable for you, that'll work in a vitualenv or on windows. And the script name doesn't have .py at the end in his example, though he doesn't call that out specifically.

I should write up a comparison between:

- Cement (http://builtoncement.com)

- Cliff (http://cliff.readthedocs.org/en/latest/)

- Plumbum (http://plumbum.readthedocs.org)

- Argh (https://pypi.python.org/pypi/argh/0.24.1)

- Aaargh (https://github.com/wbolster/aaargh)

- Baker (https://pypi.python.org/pypi/Baker/)

So many more to choose from. Now we get to evaluate Click. Seems like the reason Armin wrote Click was to load options dynamically, but that's what Cliff does via stevedore (https://github.com/dreamhost/stevedore).

My favorite feature about Cliff though is: http://cliff.readthedocs.org/en/latest/complete.html which comes out of the box, but then again, there's Argcomplete (https://github.com/kislyuk/argcomplete).

EDIT: Updating from previous posters

- Naked (http://naked-py.com)

- Docopt (http://docopt.org)

- Clint (https://github.com/kennethreitz/clint)

- Argvard (https://github.com/DasIch/argvard)

- Commandr (https://github.com/tellapart/commandr)

- Argtools (https://pypi.python.org/pypi/argtools/0.1.2)

- Plac (https://pypi.python.org/pypi/plac)

Clime (https://github.com/moskytw/clime)

cliapp ( http://git.liw.fi/cgi-bin/cgit/cgit.cgi/cliapp/ )

https://pythonhosted.org/pyCLI/#

This looks pretty similar to aargvard [1] (which is, as stated in the README, inspired by flask).

[1] https://github.com/DasIch/argvard

https://readthedocs.org/projects/argh/

I also really like argh, though it seems the provided link documents an api the mandates decorators and in fact what I like about argh is that functions can remain clean. http://argh.readthedocs.org/en/latest/

You're never forced to use decorators, they're just syntactic sugar.

Their example:

  @click.command()
  @click.option('--count', default=1, help='number of greetings')
  @click.option('--name', prompt='Your name',
              help='the person to greet', required=True)
  def hello(count, name):
      for x in range(count):
          print('Hello %s!' % name)

Could be written as:

  def hello(count, name):
      for x in range(count):
          print('Hello %s!' % name

  hello = click.option('--name', prompt='Your name', 
             help='the person to greet', required=True)(hello)
  hello = click.option('--count', default=1, help='number of greetings')(hello)
  hello = click.command(hello)

The command decorator is nice, but the group decorator feels wrong, why create a function that does nothing just to decorate it? I'd rather use `cli = click.Group()`.

The biggest gripe I have with using Python for rich command-line tool is the startup time. One of the first reason I like to write a nice command-line tool when I start a project, say foo, is to be able to do `foo --help` to quickly see and remember what the project can do (I have a very bad memory and doing this makes it possible for me to jump back faster to a project, even well documented. I can forget what I was doing in just a few days and so I add a lot of small commands).

In short running `foo --help` should be instant and if it loads all its modules to list the different sub-commands and their respective description it is really too slow.

A possibility is to cache some information (e.g. generate a text file or a small Python script).

Hm, it'd be pointless to argue this point (if you think it's slow then it's slow) but I've created many a Python CLI and have never noticed them being slow to start. Perhaps it's a case of importing certain modules globally instead of on a per-function basis?

See my answer to coldtea. Yes you can organize you're code to help making it load faster but I don't think it is fast enough. Maybe my terminal is slow, maybe my machine is slow (and maybe I'm overly sensitive to the problem) but the difference is telling.

>The biggest gripe I have with using Python for rich command-line tool is the startup time.

What startup time?

I even have a python script running on every shell prompt drawing (that checks mercurial on top of starting the python interpreter), and the latency is negligible.

I just tried on an app at work (Django app):

    > time python manage.py --help
    real	0m0.473s
    user	0m0.240s
    sys	0m0.148s

Half of a second is very noticeable (and annoying). I had similar perception in my previous work.

You can try to only load enough to display the help text without really loading everything, but that doesn't work that well (you have to organize things differently, and `--help` requires to load a lot of stuff. `subcommand --help` needs less but it still has to see if the subcommand exists).

As a reference:

    > time python -c 'print "hello"'
    hello

    real	0m0.041s
    user	0m0.024s
    sys	0m0.012s

    > time echo hello
    hello

    real	0m0.000s
    user	0m0.000s
    sys	0m0.000s

Actually even the first (0.041s) is not instant, while the second one does (I mean as I perceive it visually).

This is why I use Lua for command-line tools. Here's one such program that makes no attempt to organize code to make --help load faster:

  real  	0m0.006s
  user  	0m0.000s
  sys   	0m0.004s

Still, your second example is an order of magnitude faster than the example with Django. That's likely because Django loads the world on startup, parses all your model files, checks your database settings, etc.

To be fair, Python without loading site packages is pretty damn fast. Definitely faster than most other things loading up.

  $ time python -S -c 'print "Hello World!"'
  ...
  real	0m0.007s
  user	0m0.004s
  sys	0m0.003s

That's about the same speed it takes my system's cp command to show me an error page:

  $ time cp -fail
  ...
  real	0m0.005s
  user	0m0.003s
  sys	0m0.002s

That is your Django app, not Python itself, putting in that overhead.

Calling a shell built-in isn't really fair -- you should at least compare to /bin/echo (on my system the relevant times are [edit: 0.001s] for echo, 0.007+0.004=0.011 for python -S -c "print 'hello'". Without dropping site-packages (without the -S) python jumps to 0.014+0.012=0.026 -- marginally above 200 ms which I suppose is a perceptible difference between instant, and not-quite-instant).

Sadly, both pypy and python3 are slower than python2.7. Also worth nothing that the sys time fluctates for me -- in other words when it is > 0.00s it doesn't appear to have anything to do with the command run.

> Half of a second is very noticeable (and annoying). I had similar perception in my previous work.

That's because it has to load django and the whole django project. Here's click:

    > time python test.py --help
    Usage: test.py [OPTIONS]

    Options:
      --help  Show this message and exit.
    python test.py --help  0.06s user 0.02s system 95% cpu 0.078 total

and argparse:

    > time python test.py --help
    usage: test.py [-h]

    optional arguments:
      -h, --help  show this help message and exit
    python test.py --help  0.03s user 0.02s system 93% cpu 0.054 total

reference:

    > time python -c 'print "hello"'
     hello
    python -c 'print "hello"'  0.01s user 0.01s system 88% cpu 0.025 total

I'm completely clueless about this, but would it be possible to have a persistent python interpreter always running in the background somehow, and submit the commands to it?

If loading the rest of the code and initialization time really matters, yes. But then you're back to square one: you now have to create a CLI to access the server and that new CLI must load quickly. Which would also be the case if you just wrote it that way first, then loading the rest of the code once the command is determined.

And if you're in some project similar to Django, you would have first to query the server to know the possible subcommands...

So what you say makes sense when other parts of the code are slow to initialize and must be accessed frequently, not for the CLI itself.

You can keep ipython kernel running in background if you want. Quick unscientific test showed that it didn't reduce ipython console startup time significantly. But it should be possible to make a leaner console to connect to the kernel.

edit: my test was flawed, with more accurate test there is almost a full second time difference:

    $ echo -n | time ipython console --existing kernel-29793.json
    1.06user 0.06system 0:01.52elapsed 74%CPU (0avgtext+0avgdata 29968maxresident)k
    0inputs+64outputs (0major+19245minor)pagefaults 0swaps
    $ echo -n | time ipython console
    1.07user 0.08system 0:02.40elapsed 48%CPU (0avgtext+0avgdata 30264maxresident)k
    0inputs+72outputs (0major+20538minor)pagefaults 0swaps

of course two seconds is ridiculously slow either way

I think you want a pool of python interpreters read to go.

Ahh.. I really like pocoo's products. But I've found manage.py (https://github.com/Birdback/manage.py) to be far leaner and simple. What do you guys think?

don't like that, clashes with Django

I think this is intentional.

Django's manage command is already extensible, so if you're using Django then perhaps it's best to use its tool chain.

This doesn't look much different than argh, which has been around for ages: http://argh.readthedocs.org/en/latest/tutorial.html

Looks nice but the example looks almost exactly like https://pypi.python.org/pypi/argtools/0.1.2

Reminds me of Commandr. Both use decorators. https://github.com/tellapart/commandr

Little unfortunate naming. Isn't Click a trademark of ASF? http://click.apache.org/

Looks like a nice wrapper around argparse: https://docs.python.org/3/library/argparse.html

It's a wrapper around optparse, as per http://click.pocoo.org/why/.

I'm curious about that decision given that optparse is deprecated. It says it's because argparse doesn't allow nested commands, but I don't really see why that feature is so desirable as to warrant using a deprecated module. I probably need to have a play with it to find out.

Looks like a nice module.

Click can be extended lazily which helps execution time a lot when working with many, many plugins. It also can be extended at runtime as per configuraton.

argparse requires the parser to have full knowledge of everything which makes it slow if you add many commands. Biggest problem though is that it's parsing system is a bit broken when it comes to escaping. Options with arguments cannot have values starting with dashes which is problematic for delegating subcommands to other things.

For what I wrote Click for I could not find any alternatives that worked that way besides optparse itself but that is hard to use.

Can you provide an example of what you're saying? I've found that argparse.ArgumentParser.parse_known_args() and sub-parsers can do everything I can see click doing. Including lazy loading of plugins and the like.

Argparse has too magic of a design that you could fully nest it. I tried it many times but you always run into it's limitations.

Links to some open bugs on it:

  * http://bugs.python.org/issue13966
  * http://bugs.python.org/issue14191

Trying to figure out how this compares to Naked (http://naked-py.com) in terms of goals.

Time to rewrite some of my dirty command-line programs

https://github.com/mitsuhiko/click

user@uweb1:/$ pip install click Downloading/unpacking click Could not find any downloads that satisfy the requirement click No distributions at all found for click

do you know plac[0]?

[0] https://pypi.python.org/pypi/plac

There are like 20 libraries for this already and this looks very similar to those. But Armin Ronacher, therefore it will be popular

the name is a bit oxymoronic

is there anything like this for Go?

Can't say I've tried it but the above-mentioned docopt has a Go implementation: https://github.com/docopt/docopt.go

Reply on news.ycombinator.com