12 Factor CLI Apps | Hacker News Reader

12 Factor CLI Apps

medium.com

524 pointsdickeytk247 comments

Don't get me wrong! I love command line apps. But I wonder if we all have a bit of an Stockholm syndrome... there are several things that suck about them...

While writing this I'm thinking on my experience trying to do anything with ffmpeg or imagemagick... or even find.

* For any sufficiently complicated cmd line app, the list of arguments can be huge and the --help so terse as to be become useless. For man pages, the problem is the opposite... the forest hides the tree! I'm sure we all end up using google to look for example invocations.

* Very often completion doesn't work, since custom per-app machinery is needed. For instance: git-completion for with bash-completion.

* Sometimes I end up passing the help output through grep, then copy-pasting the flags from the output, and then hoping I got the right flag.

* ...how about things like regular expressions parameters... always so hard to remember the escaping rules! (and the regex flavor accepted by each different app).

* Not to talk about more complicated setups involved -print0 parameters or anything involving xargs, tee, and formatting with sed and cut, etc.

Is there a better way? Not sure. I like powershell a bit but some of the things I mention above still apply.

I think we may be able to get a workflow that is a bit closer to the tooling we use for writing programs while not being perceived as verbose and heavy (I'm thinking, the kind of workflow I get with a Clojure repl).

For a quick args reminder, without having to dig through man pages, try tldr: https://tldr.sh/

Speaking for myself of ffmpeg and imagemagick, I don't find the CLI itself difficult, I find that their application domain is a bit complicated for a layman.

Meaning I usually struggle to comprehend what particular options mean rather than how they bind to CLI.

Maybe indeed a good intuitive UI would give more intuition about the more obscure options. But then again, a good intuitive UI is a quest of its own.

What is difficult with ffmpeg is the sub-language in each argument, which depends on the filter you’re using. I can learn to lookup the manual, but if I have to go to the internet to learn the syntax of the filter, I’m better off googling my usecas.

And using multiple filters in one call is even more funny. Getting the output and parse it too, as it can be different in different versions and it's terribly inconsistent between filters.

Merge a separate video and audio file without looking it up - I've got a document stored that lists all the different commands I've used and hopefully some context about what they do.

> I'm sure we all end up using google

Which is kinda a CLI application.

But done the right way. With autocompletion based on your frequent searches, recent searches, machine learning based suggestions using what the whole internet searches, etc.

It's qualitatively different from your average CLI experience. As such, I don't think you can actually compare the two.

It is a command-based line-input UI. Just because it is different in other ways doesn't mean it a comparison is unwarranted.

My argument was that the "other ways" it is different in are more important than the similarities.

Powershell made 2 things gospel and I'm just sad it's going to take 20 years for other operating systems to realize that's the way to do it and something that replaces linux to see it in practice (i.e. it just won't)

1. Auto-complete is by-design part of the language/shell

1. Pre-approved Verb list that prefix commands helps discoverability and usability no end. Still learning? Get-<anything> will literally never cause a problem.

Because of the above get-help <function> can automatically create some fairly useful documentation right away and whoever wrote the code didn't need to do anything. They can significantly add to to help though. Because that mechanism is part of the language it's worth doing literally every user is going to access it, not some blog your wrote 5 years ago I hope is still online.

Passing objects rather than string, whatever maybe it's not the best way, I think it's great. But That's not the only thing PS has to learn from.

If anyone uses the cli and doesn't know how the PS help system works, it's certainly a breath of fresh air to learn.

I never really found the verb list to be entirely satisfactory. It's great that they gave boundaries but I find it too verbose. There are other kinds of syntactic ergonomics with concise vocabulary (lisp has ! xxx p$ for instance, they're a bit harder to swallow but I find the code a bit more poetic and easier to remember as a pattern).

About the objects vs strings, Kalman Reti (of Symbolics IIRC) talked [1] about how an OS passing pointers could enjoy a much easier time instead of serializing everything as strings then deserializing, especially when done ad-hoc through sed/grep/perl/whatever .. It pains me to see this. It pains me to see how linux basic utils are 30% --usage, 30% output formatting (and they all share this).

MS did a great thing with PS.

[1] https://www.youtube.com/watch?v=o4-YnLpLgtk

The verb list is, in practice, 90% Get-* and Set-* , so for most users that will cover the case where they need to read and modify some system state in a script.

It may not cover your particular use case (I want to foo this bar), but as restriction on the language it helps your users not have to discover esoteric commands.

I like it. Nowadays CLIs got out of hand with verbs, every CLI assigns a very specific meaning to their verbs and it's not clear at all. Just look at kubectl: https://kubernetes.io/docs/reference/generated/kubectl/kubec... What is the difference between "get" and "describe"? It's totally not obvious what "drain", "cordon", "taint", "scale" or "expose" do. Something like Set-KubeReplicaCount describes way more accurately what scale actually does. Or New-KubeService for "expose".

Remember that you can always use and create aliases, so you could still use kube-scale or kube-expose in the terminal if you want. But for scripts readability is the most important thing. Any newcomer can look at any PowerShell script and know what it does. And any newcomer could type in New-Kube, hit tab, and see what new things you can create in Kubernetes, instead of having to google what the command for creating a service is.

You're right, these are completely obscure, and part of the friendly neologism fad we're seeing. Please note that the examples I showed about lisps aren't short sighted concepts but very generic ones (predicates, side effects, global variables), hence the limited scope of confusion here.

I find PS tab complete to be sub par compared to fish. It’s too aggressive when there are multiple completions.

What do you mean by aggressive?

I don't know what they mean, but I know that I really dislike how PS's autocomplete works.

If the possible commands are (for the sake of discussion) `Get-AppLockerFileInformation` and `Get-AppLockerPolicy`.

If you type `Get-App` and hit Tab, it will autofill to `Get-AppLockerFileInformation` which I just really don't like, and I need to keep hitting tab to cycle through all other possible `Get-App*` commands.

What I want is some UI that when I hit tab it autocompletes as much as possible until there's a decision to make, and then shows me the options. So in my example case, it would look something like this:

Type `Get-App` -> hit tab -> shows `Get-AppLocker` -> type "P" -> hit tab -> `Get-AppLockerPolicy` is displayed

bash-ish systems nail this, and they even have the "double tab" to list all options from that point which is normally pretty nice.

The PS idea of "keep hitting tab or shift-tab while you cycle through all possible options" sucks in comparison. Especially with discoverability (There are about 20 commands on my windows system that start with "Get-App", and in order to figure them all out I just need to keep hitting tab until it cycles around)

With PSReadLine, you can change that:

  Set-PSReadLineOption -EditMode Emacs

  Set-PSReadLineKeyHandler -Key Tab -Function Complete

The first one seems to change it how I want, but the second doesn't. And it doesn't stick around after closing the powershell prompt.

And while I'm sure there's a way to make it persist, a shell's power is in it's ubiquity. If i have to change settings on every machine I work with, i'm probably going to want to use that time to install a different shell.

I know that's an attitude that is pretty impossible for any new shell to solve for, but it's the truth. I'm not choosing the most powerful or "best" shell, i'm choosing the one that gets in my way the least. And for me, that's bash in 99% of cases. And until another comes along that really improves discoverability and makes it easy to learn and adjust to, i'm not going to switch.

I have a ton of knowledge built up over the years about bash and I'm not in a position where I'd be able to dump it all and painfully try to learn everything new from scratch by reading manuals, trying out syntaxes, searching around for if what I want to do is possible, figuring out the "right" way to do things, and more.

Even if it is better in the end, if I can't get over the initial hump, it's not very useful to me. Maybe this is just me getting old...

Making things like that persist is done through the powershell profile stored at $profile. It is the equivalent of a ~/.bashrc.

Actually, that isn't a PowerShell idea.

Firstly, that form of completion has existed in the Microsoft/IBM operating system world for significantly longer than PowerShell has, tabbing backwards and forwards around a list of matches being the way that one did filename completion with some MS/PC/DR-DOS tools back in the 1980s. Secondly, PowerShell ISE can do things differently, so this cannot be a PowerShell thing.

Crank up PowerShell ISE, enter "Get-App", then press Control+Space. For more, press Control+Space to complete an option after entering "-".

I didn't mean to imply that it was a PS creation, or that it was impossible to change, only that it's how PS handles tab completion, and that I dislike it.

Being able to change defaults is nice, but the power of PS or bash is that they are ubiquitous. So anything that has to change defaults to be usable makes it kind of pointless in a lot of ways (at least to me). Because if I need to change defaults on every system i'm on, then why not just use that time to install a different shell? (which is pretty much what I end up doing on windows machines)

Not sure what version you are using, but that is not how PowerShell does completion for me. If I type in `Get-Package` for example and hit tab, I get printed a grid of the possible completions. Then I keep typing until it's not ambiguous, at which point it will complete the whole command. If there are hundreds of suggestions, it will ask if I really want to see them all.

I am running PowerShell 6.1.0 (which I think uses PSReadline). PSReadline is also very customisable, but this is the default behaviour.

I think that the solution is clear, make a language server for the command line. Each and every single command line interface is its own language so they all need their own language server that can provide meaningful completions. Then bash and other shells can add support for language servers and we have the problem solved.

I imagine this to be a little bit over the top in most cases. A standard flag like --complete-rahyeiB1 (with a sufficiently long random string to avoid collisions with existing flags) which takes the already written argument list as an argument and prints possible completions would be nice though.

> Sometimes I end up passing the help output through grep, then copy-pasting the flags from the output, and then hoping I got the right flag.

  man x | grep -P y -C 3 | less

...is my standard here (-C 3 includes the pre- and succeeding 3 lines). For all the delights of CLIs, man pages are by default opaque walls of text. And dear lord the number of regex variants... - at this point, if at all possible, I'll avoid application-specific flavours entirely, and just pipe to grep -P (perl-style regex).

I might suggest, however, that image/video manipulation is a case for which CLIs are uniquely unsuitable.

There surely are better ways (as you mentioned, Powershell appears to be a step or three in the right direction, though I have little familiarity with it), but it seems as though the greatest difficulty is in maintaining full backwords compatibility, while still encouraging new applications to make full use of novel features - not to mention the effort of modifying old applications to fit new standards that would need be expended.

If you're already piping to less, just use its search functionality; type '/', the regexp, and then enter.

For that, you might as well just use the regex functionality in man itself too. But that misses the point.

The goal of grepping in that snippet is that you're `less `ing through precisely the parts that matter, and no more, rather than wading through the whole man page, and you're using -C to control how much context you (think you) need around the search results. This is a much better setup for skimming through potential hits than going through the man page wall of text.

'man' has no regex functionality; on most Unix systems, it just uses 'less' as its pager.

sure, which means that it inherits regex support via less. You either have it on both, or neither.

You can also do that directly in 'man'.

No. All three of the responders so far have implied that this functionality is in man. It is not. It's in the pager that man invokes; ironically often, but not necessarily always, that very same less program.

IIRC you don't need to pipe to less at all for that - man already supports it. grep -C gives a good summary of the various options, then you can just pull up the man page and use that selfsame search to get details that might've been cut off.

At this point I'm not really sure what makes command-line so great. We should have something like it in the GUI sapce that works much better but, like you said, "stolkholm syndrome".

Why can't a pipeline be a more complicated multi-io workflow? In a 2D GUI this would be trivial to construct and read, but in a 1D command line it would get confusing in a hurry. And the concept works much better with AV, I can easily construct and reason about complicated arrangements of audio and video inputs and outputs, with mixers, compositors, filters, shaders, splitters, etc. between them.

Instead we worship text. Is that because manipulating text is actually more useful, or because our tools are only good for working with text?

Text is exact, programmable, repeatable and transmissible.

exact: in many GUI tools, you can have non-default settings that you changed via menus. Where are they stored? Which ones are currently active? Does it matter that you selected four objects first, then a transform tool, then another object?

programmable: > find /var/spool/program/data -name foop* -mtime +3d -print

vs "open the file manager, go to /var/spool/program/data, sort by name, secondary-sort by last modification time, find the ones that are more than 3 days old, make sure you don't slip"

repeatable: OK, do that again but in a different directory.

transmissible: here's the one-liner that does that.

Now, your specific requests are about audio and video toolchains, where I will admit that reasoning about flows is easier with spatial cues -- but I'd really like the output of that GUI to be an editable text file.

Command line is not the only HCI that can use text. Also, no one (except immense mental inertia) stops developers from producing serializable graphical interfaces.

I think the macro language of WordPerfect 5.1 was one of the most awesome serializable interfaces ever.

And they scrapped it in WP6 for some shitty object oriented version that lacked all the "serializable workflow" of the previous one.

The command line is great because it makes it easy to automate tasks involving multiple arbitrary applications. It's why we still use it even after GUIs basically became mandatory on all user-facing computers.

Yes. IOW, GUI's are not as easily scriptable as CLI programs, to compose operations from combinations of other operations, even though some tools to do stuff like that do exist, e.g. AutoHotKey and others like it.

A classic CLI command composition (a.k.a. pipeline) example:

More shell, less egg:

http://www.leancrew.com/all-this/2011/12/more-shell-less-egg...

After reading that post, I wrote about it here:

https://jugad2.blogspot.com/2012/07/the-bentley-knuth-proble...

with a couple of solutions in Unix shell and Python.

Why you should learn just a little Awk (2010) (gregable.com)

https://news.ycombinator.com/item?id=17322412

A comment by me there:

https://news.ycombinator.com/item?id=17324572

I think we (cli advocates) worship text because language happens to be a very powerful user interface.

What we need is a ubiquitous command interface where the output of any command is a tree set of typed objects and any command is free to consume any of those sets of typed objects. The typed objects in this pattern would not be limited to text.

This paradigm removes the restraints of GUI applications but satisfies a great many use cases with the obvious exceptions being any program that requires a lot of mouse or pointing device input like photo manipulation.

Anaconda and similar big data tools are moving into this space. They are for data exploration not general system tasks, but still it’s a nice higher level command line.

The Art of Unix Programming by Eric Raymond is a good in-depth look at (not just) Unix CLI programs, covering both pros and cons, some:

http://www.catb.org/esr/writings/taoup/html/

https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming

Writing is GUI takes time. With CLI you can provide much more functionality with much less time. You're free to write GUIs for all CLIs in this world, however.

> I wonder if we all have a bit of an Stockholm syndrome

I used to agree with you on this point, but then came Bender. Chatbots singlehandedly liberated the CLI from the terminal and made them ubiquitous once again.

So yeah, it's not just the terminal where we're creating CLI applications anymore.

Most people who claim that command line is the best tool for power users know nothing about history of GUIs. They think that laughable garbage that Windows calls user interface is "how it's supposed to work", and proceed to smugly lecture everyone on how command line is the only interface that can be composed and easily recorded.

(The following paragraph is not directed at the author of the parent post. It's fully rhetorical.)

Did you know that icons were supposed to be live representations of in-memory objects? That objects were more fundamental for the OS than files? Did you know that windows were views onto those objects? Did you know that interactions to and between icons were synonymous with OOP polymorphism?

https://www.youtube.com/watch?v=Cn4vC80Pv6Q

And this isn't the best UI, it's simply the first modern UI.

And how you do operate on the data in those objects? How do you filter, project, sort, stash for later, combine with data from elsewhere, edit, etc? How do incrementally build up a composition of operations from repeated experimentation? How do you record that composition in a script?

>And how you do operate on the data in those objects?

You don't. Your question makes as much sense as asking how to do polymorphism in shell commands Not operating on data was the whole point of OOP. (Or at least one of the key points.)

http://userpage.fu-berlin.de/~ram/pub/pub_jf47ht81Ht/doc_kay...

  "I wanted to get rid of data. The B5000 almost did this via its 
  almost unbelievable HW architecture. I realized that the 
  cell/whole-computer metaphor would get rid of data, and that "<-" 
  would be just another message token (it took me quite a while to 
  think this out because I really thought of all these symbols as 
  names  for functions and procedures." -- Alan Kay

The way to integrate unrelated objects in fully OOP UI would be by making them send messages to one another, either directly or indirectly. The way to store this integration for later use would be by creating, modifying and serializing an object that represents it.

Look into Pharo or Squeak, at least watch some demos on YouTube.

If your command line tool has a man page then fish shell can parse it to automatically know about command line arts for auto completion.

> Error: EPERM - Invalid permissions on myfile.out

> Cannot write to myfile.out, file does not have write permissions

> Fix with: chmod +w myfile.out

I actually much prefer:

"can't write myfile.out: Permission denied"

This shows the same information as the first 2 lines combined from the example, and the 3rd line is not necessarily the correct way to fix the problem anyway (e.g. you might be running it as your user when it should be root, chmod +w would not help).

If you are so convinced that chmod +w is the way to fix the problem, why not just do that and carry on without bugging the user?

And having each error confined to one line also means it's much less likely that some of the lines are missed, e.g. when grepping a log.

EDIT: And to add to this: it's sometimes useful to prefix the error messages with the name of the program that generated them, so that when you're looking at a combined log from different places, you know which of the programs actually wrote the error, e.g. "mycli: can't write myfile.out: Permission denied".

So, I understand the basis of your comment. You have the knowledge to know that there are other things that may be "the right way" given your situation. I think what the author is getting at is that there are users who don't have that knowledge. Giving them a hint that is verbose and non arcane can make a world of difference. Speaking from personal experience, there are many developers that I have met who don't have basic *nix knowledge, much less knowledge of a terminal. The reality of the situation is that a business is going to hire people regardless of that ability. They want someone who can move the features out the door. Whether this is good or bad is probably beyond this conversation. I think, for those users, these sorts of helpful hints are extremely important because it makes them feel like they aren't stuck and helpless. I think that, to your point, it may be useful to have the CLI offer a "pro" mode in which you could set a config to not give you as verbose error messages. Annoying? Yes. However, it would strike a balance and serve both needs.

I understand what you're saying, but this conversation is about the best way to design CLIs.

I agree that you can do lots of stuff suboptimally and still have a usable tool, but I disagree with the author on what the ideal error output looks like.

yes, exactly. I think being verbose and more human friendly is a much better way to design a tool than being terse and machine friendly.

Ultimately all we're doing is trying to save human time anyways.

In practice, "being verbose" is usually the opposite of "human friendly". I much prefer a spot-on one-line error message over dozens of lines of logs with an error somewhere hidden in them. Verbose error messages usually mean that you didn't have time to design succinct ones.

Joel Spolsky's timeless advice applies: "Users can’t read anything, and if they could, they wouldn’t want to." https://www.joelonsoftware.com/2000/04/26/designing-for-peop...

Then again, on a recent linux system, the non-ability to write to the file might be permissions. Or an immutable attr, or selinux, or apparmor, or setfacl flags or a RO mount where it lies. As soon as you decide to print out the solution to "can't write to: X" you are in for a page full of advice on what to look for. Perhaps the disk was full, perhaps uid was wrong, perhaps the 5% reserved-for-root-only kicked in. You'd end up writing a unix sysadmin guide, and then perhaps the parent dir had too strict perms to allow you to write to a file in it...

Also, on an unrelated note, I would never ever suggest novice users to blindly just give `chmod +w` to random locations. This is only marginally better than the `chmod 777 <root-folder-name>` that used to be so spread out in many (e.g. PHP-related) tutorials a decade or two ago.

Agreed, though perhaps it is just a bad example. I could imagine a situation where a single line of advice could be useful outside the realm of filesystem security or disk usage.

A few things to unpack here. First, this is just an example. A real world scenario would be something domain specific. Maybe I could come up with a better contrived example here.

Still, conceivably the app could check the file owner before showing the message. If it was a common enough error, it might be useful to do something like this.

There is a difference between an error title and error description. The description can and should be long to help clarify what is wrong. It's ok to be verbose. It's ok to spill out on multiple lines. You're much more likely to be helpful to a confused user than someone grepping logs and looking for terse output on a single line. That user can just use `grep -C` anyways.

I agree that it's useful to include the app name though.

Its easier to google a terse one-line error than a multi line error.

That's what the error code is for

iPXE is the best example I've seen of that - every error code is a link to a wiki.

IBM does that, some ANS4543 code that is the real error, _then_ a possibly translated-to-your-language error sentence that tries to help you, but if it isn't enough, you google for the error code and can potentially find help regardless of what language the blog post is in, or the forum link or whatever.

Indeed, on OS/2 one didn't google for the error code. One used the HELP command, one of whose modes of operation was looking up and printing the short and long message texts for such codes. The latter would often contain both EXPLANATION and ACTION parts.

    [C:\]help sys0003 
    SYS0003: The system cannot find the path specified. 
    EXPLANATION: The path named in the command does not 
    exist for the drive specified or the path was 
    entered incorrectly. 

    [C:\]help sys0002
    SYS0002: The system cannot find the file specified.
    Explanation: The filename is incorrect or does not exist. 
    Action: Check the filename and retry the command.

The example provides the same error information three times ("EPERM", "Invalid permissions on myfile.out", "Cannot write to myfile.out, file does not have write permissions"), followed by a recommendation that might not be correct.

Yes, this is an example, but surely you'd want an example that shows the strength of doing this verbosely? A good example of a verbose error message system is Rust's compiler output (or newer clang/gcc outputs). Being verbose for no reason other than to be verbose is just wasting the users' time (or they just ignore the spam -- which is what I would do if I used a tool that spammed me with multiple lines of output whenever it hit an -EPERM).

Personally, something like:

  % foo ./bar
  foo: write config to "./bar": Permission denied

Is clearer to me than your example. Maybe something like

  % foo ./bar
  foo: write config to "./bar": Permission denied
  Hint: Have you tried <possible-recommendation>?

Is sometimes okay (and I have done this for my own projects as well), but it's something that should be done in moderation...

> You're much more likely to be helpful to a confused user than someone grepping logs and looking for terse output on a single line.

There are two sides to this. Outputting lots of text can also cause a user to get confused (if we're talking about making things easy for not-necessarily-technical users).

When teaching (high-school) students to program, we quickly learned that even somewhat verbose output like Python's stacktraces can cause students to suddenly become anxious because there's a pile of text on their screen telling them they did something wrong. Adding more text to output does not always help, and you should keep that in mind.

It's worth considering the source of this advice. Heroku is a service for which a major customer segment is people who are not adept with unix. A large part of Heroku's power, and success, comes from the fact that it makes it really easy to do things without understand everything that's going on under the hood.

Yes, this is a clear break with the unix tradition! But it's not intrinsically bad - it's just a context to bear in mind.

> 7. Prompt if you can

Please don't. There is nothing wrong with interactive tools, but by default, they should not be. So instead of making non-interactive session possible via flags, the default should be to be non-interactive. If there is an option to start an interactive session, everything is fine.

Otherwise, you would never know when your script could run into some kind of interactive session (and therefore break; possibly after an update).

My interpretation was that the prompt is for required information. In the example graphic, "run demo" really does require that "stage" be specified. This is considered more user-friendly than simply crashing. If you don't want to see the prompt, provide that information as a flag or in a config file or whatever.

Correct, and if stdin is not a tty it should error out instead of prompting

The one thing missing from the example is a little more contextual help. Not only should it prompt you for the stage, it should say “use --stage [development|staging|production] on the command line to skip this prompt” or some such. (Could be as terse as the prompt reading “please specify --stage”.)

Like for the confirmation example? I agree. It would help clear this up. The points people are raising here with prompting are definitely not issues, they're just misunderstanding my point.

I'm not convinced all of them have even read this perfectly understandable point.

This is how Powershell treats mandatory arguments. Provide args via flag/position or be prompted for input. You can always break out with Ctrl+C if you'd rather modify the command.

> This is considered more user-friendly than simply crashing

(I'm going to assume the passive voice means "the article considers this more user-friendly" rather than some sort of commonly accepted fact).

I disagree with this strongly and agree with the GP -- I would much rather have the command exit with a message saying that a required parameter is missing. For example, if I have a script using a command and the command becomes interactive, then my script is dead; but if it simply exits then my script has failed at a repeatable point.

You could say that I should pass a "--noninteractive" flag into everything just in case, but sometimes these things aren't supported. I would much rather have an application support a "--interactive" flag to support those who want to be able to interact with the tool.

I think the two sides of this are unlikely to be able to convince each other. At least the article presents a reasonable-ish middle ground of always offering help as to the way to circumvent the interaction at the point where interaction is required.

So you want the script to [EDIT:] exit with an error code rather than hanging around waiting for input? That seems possible, with some sort of generous (e.g. 5 minutes) exit timer on the prompt. Would that satisfy your concerns? If not, what else is needed here?

ps. the verb "considered" is a good sign that this is an opinion, and would be even in a more "active" sentence.

First of all I think "crash" is the wrong word here. Unix tools commonly exit gracefully with an error code when the argument requirements are not met. Often with an informative error message.

A five minute pause sounds ridiculous to me, absolutely not user friendly from either end. It's jus unpredictable and time wasting. If you absolutely must, you can use 'isatty' to check whether stdin/stdout/stderr are connected to a terminal and act accordingly.

There is some merit to having consistent and predictable behavior regardless of where and by whom the tool is invoked, though.

Checking for the tty is discussed in TFA and in sibling comments.

User friendly until the user decides to invoke that command in a cron job and ends up with a headless process waiting for additional input.

Anyone who doesn't test cron jobs before saving them deserves whatever she gets. There are scores of ways for cronjobs to fail. b^)

Well, this is not a cronjob that failed, it's just waiting for input.

Let's say that I did test the cronjob but that it starts "failing" after an update to the tool. My fault, I know, but at least I get mail when it fails while I won't if it's just waiting for input.

This scenario would only happen if a flag became required. Prompting or not it would still be an issue. (And it wouldn't prompt as this is a non-tty environment)

> Prompting or not it would still be an issue.

Yes, but in one case the issue would result in a mail because the cron job failed, and in the other case the issue would just cause the the process to hang indefinitely without notice

NO IT WON'T HANG. I give up. I don't know how else to try to explain this to you.

It's possible to detect whether stdin or stdout is attached to a terminal, and do something different depending.

The suggestion was to prompt if stdin was a TTY and the default behavior would have been to exit with a usage error.

I think it's a fine suggestion, though in 94% of cases probably too much work to be worth it.

I think detecting whether stdin is a TTY is a huge antipattern. Most of the time, it works, but now and then, you either want to run an interactive session in a situation where stdin appears not to be a TTY, or a batch session in a situation where it appears to be. Plus, it means there's twice as much surface area to learn.

EDIT: remove rogue 'not'

You get false positives sometimes (where it claims to not be a tty but is going to the screen), but because the fall back is just reduced functionality it never causes any issues. We make hundreds of checks like this for tty in the CLI

> you either want to run an interactive session in a situation where stdin appears not to be a TTY

Um... why? That is literally the situation for which the pseudotty device was created.

There is a spectrum between "interactive" and "scripted" utilities, and the command line interfaces we're talking about sit balanced on the interface. There's no way to make everyone happy, more or less by definition. So I think "huge antipattern" is maybe spinning a bit too hard.

Prompting actually means it's no longer a CLI.

    Unless you already know your users will want man pages, I wouldn’t bother also outputting them as they just aren’t used often enough anymore.

I don't know where this is coming from, me and my colleagues are reading man pages every day. I would be interested how much others read them.

I strongly disagree on this one too.

That's the first place I look for help and it annoys me to no end when a CLI program that doesn't come with one.

I stopped taking seriously the article at that point and quickly skimmed through the rest of it.

Man pages are a great unix culture heritage, please new developers don't give up on them!

I myself also love reading man pages, but speaking of compatibility, I have to say that “--help” is a more universal way of showing help pages. Of course it’s better to have both of them though.

--help is fine, but almost never a substitute for a full man page, except for the most trivial of applications (unless your --help is as complete as a man page, in which case... good on you for providing full documentation, but I'll hate you a bit every time I unthinkingly drop two hundred lines of text in my terminal.)

Unless it just opens the man page if it is so long. Like git.

I strongly suspect that --help is on the contrary less universal, given that there exist entire toolsets lacking the --help convention but having manual pages. There was almost a quarter of a century's worth of Unix tools that developed before --help was invented.

* https://unix.stackexchange.com/a/416796/5132

* https://unix.stackexchange.com/questions/207136/

How can you say it's less universal when it doesn't work on windows?

Moreover, man pages serve as an essential resource that is much more reliably available than any other - when my network's routing packets in circles, the database server's on fire, and the duplicate has repeated the last transaction on every record, I don't care how "most users" don't use man pages or that web documentation is much more Google-able; I care that I have access to complete, detailed information (not the abbreviated version provided by --help) on all the utilities available to me; I care that apropos can help me recall the names of utilities I don't remember (something the article does not address); I care that I don't have to worry about having access to documentation, in addition to everything else, because it's all in a single, standardized, effective repository of man pages.

I really can't express strongly enough how much I disagree with the view that man pages are no longer relevant or necessary.

man pages are the first place I'm looking in for flags for commands I know for years, or when I'm using new commands — and it's always a frustrating experience when they don't exist.

Please, always ship man pages with whatever you write. It's /easy/ to do and has a great added value.

PS: doc on the web is so often irrelevant… either too old, or too recent — and in the rare cases where it's properly versioned, the workflow is something like “cmd --version ; google cmd $version” instead of just “man cmd”, which is nowhere as convenient or reliable.

As you're getting mostly replies agreeing - I never read man pages. I look up the odd parameter every now and then though.

Do you mean you look up parameters somewhere else, e.g. in a web browser using google?

Pretty much, yes.

> I don't know where this is coming from

From people who don't know how to use man pages.

BSD manpages or Linux manpages? There are very few Linux manpages that I'd consider acceptable.

When man pages are good, they're good. When they are bad, they are super verbose, incomprehensible, sparse, or otherwise completely useless.

I never read man pages. If I need to check man, the tool failed in being user friendly. I'll then rather search online for what I need and blindly copy the first result on some stack exchange page..

If you can't immediately figure out how to properly set up and adjust a contractor's table saw, do you blame the table saw for being developed for professional contractors instead of newbies?

User friendlyness is a feature, and like all features it's not always worth adding to professional tools.

Nix commands show their man pages as their help, e.g. `nix-shell --help` is equivalent to `man nix-shell`. I think this works well, and reduces the burden of writing the same documentation over and over.

Whether they're good or not, i feel manpages should ALWAYS be included every cli app. That being said, it is quite often that i begin my inquiry with manpages but then have to unfortunately look elsewhere for more info...specifically either a nice set of usage examples, or a bit more background on the "why" to use the cli app - in this order.

> I would skip man pages are they just aren’t used that often anymore.

I understand that man pages might represent a minority, but I cannot express enough how wonderful it is to get the full manual of a program without interfacing with the web. Not to mention how powerful that is, since most apps have short names that are difficult to search for, but how accessible that makes the application.

I agree that man pages are wonderful. This point in the article really irked me.

The fact they don't run on windows means some subset of your users cannot even use them if they wanted to. Better to spend your time on something they can all read.

I'm not saying they're not useful. If you've got plenty of time to write up docs, go ahead, but the reality is we only have so much time and I think we should spend our time writing in-CLI docs and web docs before we start man pages.

Also, you don't need web access to use in-CLI docs either, and that works on all platforms.

Having said this, I do plan on having man pages be an export type of the oclif docs (which is currently in-CLI and markdown). I intentionally made the output very similar to man pages already so it should be relatively easy to do.

For man pages you could suggest writing markdown and using a build process to automatically generate man pages [1] in the event they aren't using oclif. EDIT: I think we've all been in areas without network access like on a plane and not having a man page in that scenario is very annoying.

Also, you briefly say a few things about CLI apps using a remote API, you may want to add to that and say a few things about the proxy environment variables [2]. These are indispensible for corporate users. I think some early, early version of npm didn't respect the no_proxy environment variable, and for the http_proxy and https_proxy it required some arcane combination of: proxy in a flag, proxy in a config file, proxy environment variable set. It really should be an OR not an AND...

Last but not least, another annoying thing was tools changing their config format or location. I think it was docker that changed their config file format and/or location like two or three times. Absolutely infuriating.

1. https://rtomayko.github.io/ronn/ronn.1.html

2. https://wiki.archlinux.org/index.php/proxy_settings

Oh believe me. I'm intimately familiar with proxies and CLI apps. I think that might be a good standalone article though as it only applies to CLIs that connect to APIs whereas this article is intended for all CLIs, period. (Minus that note about the user-agent I suppose).

We get away without using any config files in the Heroku CLI which is certainly preferable. (Well, there is a config file, but I don't think anyone's using it and it's undocumented. I think all it can do is disable colors) Config is another topic that I do think would warrant its own article as well. I may not be the best author though as we've tried to avoid config. (Though it's a common enough problem I do want to solve generically as possible in oclif).

As far as automatically building man pages, I still think that's a wasted effort. Nobody has ever asked for or even mentioned man pages in our CLI. Setting up a build process and distribution is considerable effort and maintenance burden.

Of course if the users of your CLI want man pages then of course build them. In my experience though, that's not what users want. Though it's important to note that a CLI that interacts with a cloud service is pretty useless without internet.

If no internet is the only compelling reason to support man pages, I'm still not convinced it's a better use of your time. The docs should already be available offline in the CLI itself.

Well, PowerShell has the Get-Help cmdlet that is somewhat similar to the *NIX man command.

See https://docs.microsoft.com/de-de/powershell/developer/help/h...

For people that like man pages (there appears to be lots of you) do you think that man pages are more important than web or in-cli docs? Or just that they should be written in addition to and not missed out on?

My (current) position is that they're useful, but not worth the extra effort for most CLIs. It's a cost-benefit thing.

I'm genuinely curious as I've never had anyone request man pages in our CLI.

I for one prefer actual manpages than online documentation. The web documentation is cumbersome to find, forces me to open a browser window, and I cannot copy-paste easily from inside my terminal.

Notice that if you already have help, you can build the manpages automatically from them using "help2man". You could get manpages for all your tools by simply adding a line into your makefile!

If the man pages are simply generated from the help output, why bother having them when you can just use `--help`?

Because the expectation on UNIX-like systems is that I can type `man foo` and get the manpage for `foo`.

It's good manners for a CLI app you want installed on someone's system to also integrate with the help infrastructure of that system.

apropos

The end result argues against using help2man and thinking that --help and the user manual should be the same thing, not against having a user manual at all. (-:

* https://news.ycombinator.com/item?id=18174025

> do you think that man pages are more important than web or in-cli docs?

Yes.

* Web docs are a problem because I don't always have access to the internet when trying to do something on my computer, and usually there are so many kinds of web doc generators that you have to figure out how the information you want is laid out. Web docs are useful as a quick-start guide or a very lengthy reference guide -- but not for the common usecase of "is there a flag to do X?"

* In-CLI docs are a cheaper version of man pages. In most cases, the output is larger than the current terminal size so you end up piping to a pager (where you can search as well), and now you have a more terse version of a man page. Why not just have a man page?

Man pages are useful because they have a standard format and layout, provide both short and long-form information, and are universally understood by almost anyone who has used a Linux machine in the past. "foo --help" requires the program to know what that means (I once managed to bootloop a router by doing "some_mgmt_cmt --help" and it didn't support "--help" -- I always use man pages now). One of the first things I teach students I tutor (when they're learning how to use Linux) is how to read man pages. Because they are the most useful form of information on Linux, and it's quite sad that so many new tools decide that they aren't worth the effort -- because you're now causing a previously unified source of information (man pages) to be fractured for no obvious gain.

I still add support for "--help" for my projects (because it is handy, I will admit) but I always include manpages for those projects as well so that users can actually get proper explanations of what the program does.

> I'm genuinely curious as I've never had anyone request man pages in our CLI.

Honestly, I would consider not using a project if an alternative had man pages (though in this case it would be somewhat more out of principle -- and I would submit a bug report to bring it to the maintainers' attention).

> I still add support for "--help" for my projects (because it is handy, I will admit)

Some applications (e.g. Git) make "--help" redirect to man. What do you think of that?

Personally, I still pull up "man git-pull" or similar. I'm actively annoyed that I have to remember that the AWS CLI is different in this regard.

Not to mention that using "--help" for man pages requires I open up a separate window when I typically just want a quick reference to the most used flags.

Moving man pages to a different command is like coming up with an alternative icon to the hamburger menu for your regular UI. Sure, all the functionality is still there, but it requires a full stop and search to remember where to find it.

I prefer a well-written man page to any other information. Git (and a few others) do open the man page when invoking `--help`. That's a way to avoid some overhead.

I hate when programs do that. When I type "foo --help", I want concise help message for foo. If wanted foo man page, I would have typed "man foo".

man pages can include concise usage examples as well.

Well, man pages are the standard way to get documentation. Sometimes --help is detailed enough to be a replacement (in which case I can tolerate the lack of man pages), but usually not.

I don't really care about documentation on the web. In fact I'd rather you simply treat the man pages as the single source of truth and put the man pages on the web. Sort of like https://linux.die.net/man/

https://manpages.debian.org/ffmpeg (or whatever package) is good, too.

In terms of usability web docs are better. But having the docs on the web is not a good thing; there are many advantages to having them locally.

I'd be just as happy with bundled HTML documentation.

I also prefer manpages over web searches for multiple reasons:

- works without internet: very important when you want to use a long train ride to write some code (I also have the entire rust-doc and all IETF RFCs on my disk for quick referencing)

- quick and reliable access to known items: `man ascii` is way quicker than finding an ASCII table on the web (probably on Wikipedia). And finding the syntax for an obscure bash feature is way easier when your search is confined to `man bash` rather than to the entire web.

- Don't know how to label this, but I like that the manpage is a complete documentation of one tool, unlike a disconnected set of Stack Overflow questions. That allows one to cursory read through the manpage to learn the scope of what that tool can accomplish.

One more reason - the man pages will be specific to the version of your tool. My man pages on my mac will automatically pull up the BSD version of "ls", not the GNU version.

I see no benefit whatsoever to web docs over man pages. man pages are immediately available, where you are (the CLI) without an internet connection or a web browser.

I expect the -h flag to give me a summary of the flags and arguments, to remind me of the particular name of the flag I'm missing. I most certainly don't want the whole documentation there, partly because the whole documentation is (presumably) large enough to scroll my history off screen.

So, yes, man pages are definitely more important than web or in-cli docs.

In my limited experience, manpages give you a wall of text when 9/10 times you just want a oneliner example of how to do something. Web resources generally address the lack of real world examples in manpages.

Then you use your pager's search functionality and go to the EXAMPLES section. If there is no EXAMPLES section, that's not the fault of the format, but of the author. Presumably, the same author would be equally good (or bad) at providing examples no matter if the target format was a man page or a web page.

It's an argument that Daniel J. Bernstein made when proposing slashdoc. So M. Dickey is not alone.

* https://cr.yp.to/slashdoc.html

I myself write doco in Docbook XML

* https://news.ycombinator.com/item?id=15779321

generate HTML from that that can be read directly

    xdg-open /usr/local/share/doc/nosh/machineenv.html

or on the WWW

* http://jdebp.eu./Softwares/nosh/guide/machineenv.html

and generate roff for man pages from it as well

    man machineenv

Whereas --help output is auto-generated from the option definitions given to the command-line parser, much like libpopt does.

I almost stopped reading at "I would skip man pages", but the rest of the article was mostly great advice.

I disagree about 11 (using "main_command sub_command:sub-sub_command" rather than "main sub sub-sub" syntax), but it's mostly a matter of taste.

Seriously, though, if you've already taken the time to write documentation, then there's no reason not to also generate a manpage. Just using pandoc to convert your, say, README.md gives good-enough results:

pandoc -s -f markdown_github -t man -o your_cli.1 README.md

(There probably are other good conversion methods.)

Why I like man:

Advantage over online docs:

It's offline and available directly in the terminal, without having to open a browser and it has a distraction-free, clean look. The only slight disadvantage is the lack of support for images, which are occasionally helpful, but in a pinch, for some use-cases, you can have ascii diagrams.

Advantages over "--help":

1. Conventionally, "--help" just provides a brief rundown/reminder of the options, so having full documentation is valuable.

2. If "--help" provides the full docs then:

a) You lose the option of having the brief rundown, which is also very valuable.

b) "man command" is slightly faster than "command --help" :p (yes, it is a slight pity that accessing the full docs is faster than accessing the brief version, if you use convention).

c) man deals with things like having nice output, with proper margins, at different terminal widths.

d) man deals with the formatting for you, providing consistency with all other applications.

FWIW I think that texinfo is (mostly) even better than man, as it considerably improves on the navigation, but it's been crippled by the FSF-Debian GFDL feud, which meant that the info pages weren't actually installed on many systems, and it's mostly a lost cause now.

> Advantages of man over "--help":

You can have the best both worlds if the manpages are built automatically from the "--help" output (e.g., using help2man). Then you can have "-h" give a brief rundown and "--help" give the full docs.

> FWIW I think that texinfo is (mostly) even better than man, as it considerably improves on the navigation

I am curious about that. Do you really like texinfo navigation? I find it completely unusable, to the point of prefering to download and print a pdf from the web instead of opening (gasp!) the dreaded "info" program.

I use info browser from Emacs and like it very much. The best benefit is that you can stuff a whole book into info pages - and projects using info usually drop their full manual in there, to be perused off-line and distraction-free.

How do you search for a word inside the whole info documentation of a program (say, gcc), and cycle through all appearances of that word? I never managed to do that (which is trivial for manpages).

Don't know how it works in regular info browser; in Emacs's info browser, incremental search can cover the entire manual (or even all info pages) if it fails to find a phrase on the page you're currently viewing.

It does, see `info '(info) Search Text'`.

Typographical quality of automatically generated manpages is usually very poor.

Also, I expect man pages to be more detailed than --help.

You could have a `--full` or `--verbose` flag on the help command to display the full output. That will work on Windows as well.

One style that I've seen that I really like is having the help system be its own first-class subcommand. E.g., just `p4 help` gives you the quick summary with a list of available subcommands, then something like `p4 help filelog` gives you the details on just the filelog subcommand. I find that this avoids the problem of a having a single help spiel that is either too brief or too verbose to be useful.

(The article sort of touches on this with mention of `mycli subcommand --help` as something that should show help and `mycli subcommand help` as potentially confusing `help` with an argument. But I find that making help a full subcommand tends to avoid avoid this ambiguity. And having the bare help command give the table of contents lends structure to the help system.)

perhaps I should've called this out explicitly, but yes, I would expect a CLI to do this.

I have future plans for oclif to take this a step further and make the help contextual based on what you're working with. For example, if you wanted to see what commands might relate to a file you might get different commands than a directory.

> 12. Follow XDG-spec

I'm so glad to see this included. I don't like $HOME being cluttered with .<app> config directories, but worse than that, far too many when releasing on macOS say Oh Library/Application\ Support/<app>/vom/something is the standard config location on Mac, so I'll respect XDG on Linux but on Mac it should go there. No! Such an unfriendly location for editable config files.

I am having an impossible time verifying this, but I recall reading that MacOS deletes unaccessed files eventually from `~/Library/Caches/*`. Which would be a compelling reason to use that for cache. (Not being able to verify this I didn't add it to the article)

If anyone can verify that I'm either right or wrong here that would be helpful.

OSX does not automatically delete from that directory. See: https://developer.apple.com/library/archive/documentation/Ge...

> Your app is responsible for cleaning out cache data files when they are no longer needed. The system does not delete files from this directory.

However, many third-party tools delete from that directory with minimal caution if any, so it's a good idea to consider it ephemeral.

appreciate the clarification!

I agree that seeing a bunch of ~/.<app> directories is annoying, but at the same time I do think that it makes sense for each application to manage its own hierarchy, rooted under e.g. ~/.apps/<app> instead of splitting it into ~/.config/<app>, ~/.local/share/<app>, etc.

Regardless, I think it probably makes sense to have a uniform interface for getting said directories, so that however the OS decides things should be laid out, the developer just needs to `local_config_dir(app_name)`. If the user (or at least administrator) can decide between <app>/<function> and <function>/<app>, all the better.

The reason ~/.config/app is superior, is that then you can e.g. backup all your configs, or remove your cache, or store ~/.local and ~/.cache on a local fs and the rest of ~ on NFS.

Do you have to use ~/.config/app/ or could you just use ~/.config/app.conf?

Regrettably, OSX recommends the use of the Library/Application Support directory, and provides several utility APIs for getting config/cache/data locations which point there as well: https://developer.apple.com/library/archive/documentation/Ge...

In general, if I'm targeting OSX, I use those locations as recommended, but make symlinks to relevant config/etc. from there to the XDG-suggested locations.

Erroneously naming it "XDG-spec" is not very good, though. It is the "XDG Base Directory Specification", just one of several XDG specs.

* https://freedesktop.org/wiki/Specifications/

* https://standards.freedesktop.org/

Yes, also it should be in the top half, #12 makes it sound not very important. Certainly before "be fancy."

This is all great advice.

The one thing this does miss is distribution, which is a HUGE part of offering a great CLI app. Specifically, I'd say:

1. Make your OFFICIAL distribution channel the primary package manager on each platform (ex: on Mac, homebrew. Ubuntu, apt/snap). Beyond that, support as many as you have capacity to.

2. Also offer an official docker image which fully encapsulates the CLI tool and all of its dependencies. This can be a great way to get a CLI tool loaded into a bespoke CI environment.

yeah I agree 100%. Distribution of CLIs is a headache for the maintainer, but it's really important to get right if you want users to have a seamless experience.

Homebrew is NOT the primary package manager on Mac and I wish people would stop perpetuating that falsehood. Apple includes pkgutil/pkgbuild in the OS and that official package management strategy plays much better with corporate IT control of managed machines.

In my experience, Homebrew always eventually results in pain and complex debugging and it's almost impossible to audit software it installs to prevent the installation of prohibited or dangerous software.

It's really not that hard to build a .pkg file and developers that want to properly support the Mac platform should go down that path before offering Homebrew support.

Well, I'll admit this is the first I'm learning of this tool. That said…

The top hit on Google for "os x pkgbuild" is a link into Apple's documentation that 404s. (Further Googling turns up some blog posts, and man pages, so that's good.) Does this support dependencies? How do I get updates to users? How does a user receive updates?

Wow...I'm not sure why Apple has taken down it's documentation for the command-line tools. It's been so long since I looked at anything beyond the man pages, I didn't realize the web docs weren't up. If I had to hazard a guess, I'd bet they're trying to move developers towards an XCode-centric approach. But there are plenty of other tools that can generate that format too. And it's the only format that's natively understood by the OS, so I still stand by my assertion that Homebrew is not the official package manager for Mac.

It might not be the official package manager, but what was claimed is that it is the primary package manager. I’d wager that Homebrew is more used than... I don’t even remember what you mentioned as the other one as I’ve never heard of it before and have never used it.

The best part is that some people think 'brew' is a solid package manager and then show up in Linux and try to make 'linuxbrew' happen (no really, it is a thing).

I just wish everyone would take a day and read an intro to nix/nixpkgs and the world would really be a better place. There are so many "popular" hyped tools these days that can barely do a fraction of what is going on in the Nix ecosystem, but it doesn't seem to get the hype that brew, buildkit, linuxkit, etc all seem to get.

> hype that brew, buildkit, linuxkit, etc all seem to get

I'm not that much in devops/containers guy, bit I'm not aware about linuxkit alternative in the Linux world, care to elaborate?

As my comment implies, nix and nixpkgs can do what `linuxkit` does (in terms of image building, at least) but better and in a more powerful manner.

I can create nix derivations that look like a linuxkit yaml, but instead of having a bunch of opaque sha256 hashes to some container in them, it has symbolic references to packages that are defined in my nixpkgs repository. This nixpkgs repository includes package definitions for basically everything in a distribution. From it, out of the box, you can issue single commands to build: VM images, container images, images ready to deploy on GCE/AWS/Azure, all from a single set of package definitions.

This means I can issue a single command that will output a VM image (or a container) that includes the exact revisions of all of my software, down to the kernel options and compiler flags. It makes it trivial to take in patches for critical components and rebuild a base image. No cloning extra repos. No build, container build, push, grab sha256, copy sha256 into a yaml file. Just specify the patch, hit rebuild, done.

You can do this for VM images, specifying the total system configuration - how you want etcd/kubelet/etc running, for example. One command and you have a bootable Azure VHD. You can then use the same tree, or maybe a different branch, and declaratively(!), very minimally [1] build the most optimal container images that you then have deployed to Kubernetes or wherever.

And you can be sure that you can build this exact configuration in 1 year, 2 years, 3 years, etc, due to how Nix works.

I hope I've done a somewhat okay job of explaining this. I'm trying to take some time and write a "container oriented look at why Nix is cool" guide soon too.

[1]: https://grahamc.com/blog/nix-and-layered-docker-images

Great, thanks!

Say what you will about Homebrew, but we need more package managers that can easily install without root. Not everyone has the time and energy to compile from source, and often it's a circular problem - I need to compile and install Python 3.7, which needs openssl, which needs to be compiled from source, which has even more dependencies... ad nauseam.

Why do we need more of them when we already have a number that are capable of deploying into a home directory? As someone who has worked on software that has needed packaging, and someone who tries to help out with packaging for a distribution, I can't imagine why we need more for the sake of having more.

Per my original comment, nix can do this, for example and already has an enormous number of packages packed, pre-built/cached, ready to go.

App Store is the primary package manager on Mac (which can be used from CLI). Homebrew is the defacto standard complimentary package manager on Mac. Whether you like the fact it is the defacto standard or not is irrelevant.

> App Store is the primary package manager on Mac (which can be used from CLI)

How so? I’m not aware of anything built-in that allows for this.

This can be done using the mas-cli. It's not official, but works decently well.

https://github.com/mas-cli/mas

mas-cli uses private API to do this, so it's not like this is something that Apple encourages or provides ready access to.

Thank you for your opinion. I'm going to continue using homebrew and encouraging developers to support it.

Why would you put a CLI tool in a docker instance? I'd understand if it were a server environment or web application.

> This can be a great way to get a CLI tool loaded into a bespoke CI environment.

Okay, you have the CLI tool in one or more isolated Docker instances. What then?

No, you start the docker container to run the CLI tool, because the tool is exported as the CMD for the image, like this one [1]

[1] https://hub.docker.com/r/mesosphere/aws-cli/

I had no idea you could run a Docker image as a command. I thought it simply allowed you to run light weight VMs running on the same kernel.

I disagree on the 2nd point. Flags prevent globbing by the shell and make the help text less clear.

Consider a usage line like this:

    prog <user> [password]

This tells you which argument is mandatory and which argument is optional in a second, without searching for the help text of --user and --password.

Also, an example like this:

  git add <pathspec>...

Tells you that git add accepts multiple paths and you can invoke it with:

  git add src/*

What's more important in my opinion is making sure your argument parser can handle values that start with a dash and respects a double dash to stop the option parser.

Consider an interface like this:

  prog [--rm] [--name <name>] [args...]

And you invoke it like this:

  prog --name --rm -- --name foo

This should result in:

  {
    "name": "--rm",
    "args: ["--name", "foo"]
  }

Getting things like this wrong can result in security issues.

I may try to expand on this in the article, but it's in there if you read between the lines. There is a difference between something that takes in multiple args of the same type and multiple TYPES of args. I'm arguing against multiple args of different types, not the same.

By definition any CLI that accepts variable args is fine here as it's all the same type.

The -- is a great point as well. It solves a lot of problems users have but a lot of time people don't even know about it. It solves issues with `heroku run` for example.

EDIT: updated to clarify my point

I don't like using arguments for key/value data, e.g. `prog --name foo`. Much better to use env vars like `NAME=foo prog`, since the OS handles the mapping automatically, reading them out is trivial in every language, passing args to subprocesses is easier, etc.

> The user may have reasons for just not wanting this fancy output. Respect this if TERM=dumb or if they specify --no-color.

Or if they specify `NO_COLOR`[1]!

[1]: https://no-color.org/

yes, adding this

I'm probably being picky, but I would also include that the CLI be a self-contained binary. I'm tired of managing Python / Ruby / Node versions.

Its bad both ways. The advantage to python/ruby for example is you can simply pip/gem install, or update. With a binary, you have to download, move, and change permissions every update. For experienced linux users, the binary is fine, but for newer users, its much more "friction"

IMO using a language's package manager to install applications is a massive anti-pattern, that should be handled by your OS package manager.

This is what I prefer as well. Let me use my OS's package manager for managing my packages.

That would still be the case if the packages weren't 1-3 years out of date.

That is true. But there are ways around that. Including documentation on how to build it locally is pretty standard. And hosting prebuilt binaries with package installation for targeted platforms is also pretty common as well.

With interpreted languages with language-specific package managers, you have to:

1) Install the language

1a) Possibly have to install a language version manager (rbenv, pyenv, etc)

2) Install the language's package manager

3) Install the CLI utility via the language's package manager

Here's the order I think CLI maintainers should strive to making their utilities available:

1) Install via OS package manager

2) Install via prebuilt release with OS-specific package, from hosting site (GitHub, etc).

3) Install from source

4) Install via language-specific package manager

5) Install via curl | sh :)

The problem is, its much easier to support one language package manager, then it is to support 10 different OS package managers, not to mention, some, like debian, are near impossible to push things too.

Only a web programmer could believe that man pages "just aren't used that often". It drives me nuts when compiled cli programs don't have a man page. It says to me that the author of the program doesn't know Unix conventions or doesn't care enough to put the effort into meeting his or her users where they are, and so I'm gonna have to be careful about how I use the program lest it do something unexpected. Use man pages.

The awscli is just terrible in this respect. There's no man page for 'aws' so I say "aws --help". It then literally tells me "To see help text, you can run: aws help". OpenShift's 'oc' sucks at this only a little less, with no man pages and for some inexplicable reason you can only get a list of global options in a dedicated global options help subcommand instead of at the bottom of every help page. The documentation system for 'git' on the other hand is a work of art. Pure beauty.

I don't agree with 7 and 8. I like silent apps while working, and actually I'm used to applications saying nothing if everything is correct. Also, "outputting something to stdout just because I can" kills scriptability a lot.

Using tables, colors and other stuff requires a lot of terminal support. MacOS terminal, iTerm, Linux terminals supports a lot of stuff, but not always (our team is generally using XTerm for example). Implementing these are acceptable if there's a robust code detecting terminal capabilities and falling back gracefully and without treating these more streamlined terminals as lesser citizens, and this requires a lot of development, head banging and maintenance. If you're accepting the challenge, then go on.

BTW, That unicode spinner is nice. Very nice.

You can just have a "quiet" mode for scripting. Or even better, detect if you're connected to a TTY.

Checking whether you're connected to a TTY is a good practice, but there's also cognitive advantages of silent applications too.

Also, similarly you can have a -v --verbose set to make the application talkative. It's also an option.

I run scripts from a tty all the time...

> Also, "outputting something to stdout just because I can" kills scriptability a lot.

The article does specifically recommend using stderr for messages to the user, is there any reason that would kill scriptability?

I missed that. It's actually a much bigger problem. stdout and stderr(or) is two output paths to explicitly diverge error logs and problems from standard output.

stdout / stdin is reserved for user interaction, informational messages (md5sum), actual output (ls, less, etc.), and the like. OTOH, stderr is explicitly for error messages only, and it's very useful (and necessary).

A real use case from my daily job: I'm an administrator of a large system (approx ~1K servers), and I have substantial amount of cattle, and a lot of pets [0]. All of these servers have cron jobs, and other automated tasks on them. All servers can mail to a local mail server to report us problems, so our cron automatically mails any outputs to us.

All the tools we use, and scripts we write have the following properties:

- If everything is OK, they are silent by default.

- They output to stderr, if anything notable happens.

- Also we copy (think tee) all stderr to their respective logs (both local and on a centralized server).

Now consider:

- The (e-mail, log) noise if all the tools were writing their outputs to stdout.

- The work required to silence all tools. What if they don't have any --quiet switches?

- Furthermore, they wrote everything to stderr. How can I know whether thing has worked as it should?

- How can I find the problem quickly if everything is written to "Error" log?

- Furthermore how can I revise the errors happened quickly? Info & Error on the same file. grep galore!

- How can I silent a tool (by redirecting to /dev/null) if both normal and error logs are written via stderr?

These are the simple problems that I can come up on a real, big production system in five minutes. I can find more problematic scenarios which will happen on a daily basis, if I think more.

All these conventions, folklore, recommended usage and facilities are in place because of the needs and the experience acquired in the history of these systems. Running around them amok, just because they enable color, pretty spinners, or justify some narrow usage scenario is not correct.

These conventions and philosophy [1] allowed *NIX systems to scale without needing excessive administrative elbow grease. Ignoring these, and developing tools which use facilities and conventions as they please will degrade the ability to manage these systems with minimal work. I can always grep, but it will be inefficient and not guaranteed to get everything that I want, and also it will cost me and the computer time to do so.

Like algorithms, systems are easier to manage when "n" is small. When "n" gets big, these tasks got really hard, really fast.

So, develop tools to ease tasks. Not to show-off.

[0] https://www.engineyard.com/blog/pets-vs-cattle

[1] https://en.wikipedia.org/wiki/Unix_philosophy

Edit: Tried to increase readability.

These are all good points, and I wish more clis were like this. My own pet peeve is un-disablable stdout logging.

> It’s important that each row of your output is a single ‘entry’ of data.

It felt weird to me to use `ls` as an example as it's not immediately obvious it adheres to the advice from the printed output. I suppose they were also trying to highlight the earlier point of differing output format depending on whether output is a tty/pipe.

Unrelated, but I didn't know `ls` was that smart about isatty. Once upon a time I read the '-1' option to print one name per line in the man page and assumed it was necessary for that functionality. Thanks!

I just picked `ls` as it's a common utility everyone understands and isn't some contrived example using `cat`.

I am going to add a note about `ls`'s behavior with isatty. It's sort of conflating a couple of things, but I think it's interesting enough to leave it in.

I agree, it is a good example, and hey I learned something.

I really appreciate that you're taking the time to respond to all the feedback in this thread and wade through everyone's nitpicks. Looking forward to more articles.

Thank you for the great feedback!

> My own pet peeve is un-disablable stdout logging.

I really think that we need a stdmeta file descriptor. See my comments at the link below, I would appreciate your feedback:

https://unix.stackexchange.com/questions/197809/propose-addi...

I like the idea in Common Lisp - beyond stdin, stdout and stderr, it also specifies bidirectional streams for general interactivity (querying users for data and accepting their input), interactive debugging and an output stream for tracing&timing code. See: http://www.lispworks.com/documentation/lw61/CLHS/Body/v_debu....

Unfortunately, if using CL to deploy a CLI app for modern systems, all of this has to be shoehorned into the usual stdin/stdout/stderr split.

Man I wish OpenVMS was still a thing. All the commands worked the same due to the DCL enforcing it. https://en.m.wikipedia.org/wiki/DIGITAL_Command_Language

Non-mobile link:

https://en.wikipedia.org/wiki/DIGITAL_Command_Language

Me and you both. OpenVMS got so much right back in the day. Good error conventions, versioned file system... I still fondly run an OpenVMS workstation under my desk so I can revisit the good old days sometimes.

Then realise my skills have atrophied to the point I can barely remember how to use edit.

A) PowerShell was inspired by OpenVMS DCL. B) OpenVMS on x86 is due to be released in 2019 (it's in private beta right now).

Also on Color. Don't got go all pschadelic. I've found that some programs using 256-color is unreadable with my terminal colors.

Also unix commands tend to have illegible colors in Powershell on Windows. (Ripgrep for example). Powershell defaults to a blue background.

Yeah if I get some time I should expand on that. There are some colors that don't work well at all for common scenarios (solarized, windows). Some of them we've blacklisted and do not use.

Also for colors it's good to try to pick a decent palette of a couple-three colors and stick with it rather than try to categorize a bunch of different bits of output on a single command.

Still, colors are awesome and make your CLI look and feel 10x better than it is, so it's worth the extra effort.

A problem with color is that people end up optimizing the aesthetics for their own terminal setup. There is a wild number of different color schemes out there and it's really hard to make something that looks good on all of them. In my experience the only safe choice is bold text (i.e. \033[1m) since it stands out in all cases.

red, yellow, green, cyan, and magenta are pretty safe. We've had complaints about others, but these are pretty reliable in my experience. dim is safe too, but doesn't work all the time. (Not working meaning just not dimmed)

I would definitely not call yellow a safe color. I use a terminal with a white background, and yellow text is unreadable unless the text is also given a custom background color.

Can you suggest a better default color configuration for ripgrep? We actually already have different default colors for Windows as opposed to unix.[1]

[1] - https://github.com/BurntSushi/ripgrep/blob/acf226c39d7926425...

After some time of experimenting with this. I probably can't recommend any colors.

The issue I've found is that of the 16 build in colors. cmd defaults to trivial colors. (eg Blue is 000080 and Bright blue is 0000FF.)

Which give terrible contrast. MS seems to be working on improving things on their end. Then I'll be able to make it readable. (Eg: https://github.com/Microsoft/console/tree/master/tools/Color...)

I am one of the handful of people to use a white background in my terminals. A significant amount of coloured output is illegible to me.

I would suggest these rules for using default coloured output:

1. Don't.

2. Really, don't. Bold is fine, though!

3. (experts only) Make sure the colour scheme works with white-on-black, black-on-white, white-on-navy (for powershell), and Monokai/Solarized/whatever the flavour of the month for insecure hipsters is.

If you use colours, by default or not, make it really easy to configure the colours, so people can make it work with their terminal's colour scheme.

#6 is great, except if it causes performance issues: https://github.com/npm/npm/issues/11283

Speed is the ultimate fancy enhancement ;)

Even if it does not cause any performance issues, I dislike it. If the thing runs in terminal, it should be expect to be used in a script thus it would be better to make no assumptions of terminal capabilities and leave the fancy part to external tools, if one is interested. I always hate systemctl's piping to a pager by default. "Do one thing, do it well", don't try to surprise users with fanciness because at work we don't like surprises.

In practice I've had overwhelming feedback praising our use of spinners in the Heroku CLI and not a single complaint I can think of. In fact, I've had more praise for adding spinners and color than any other change we've put in over 4+ years of development.

That said, you need to be careful. Don't use a spinner if it's not a tty or TERM=dumb. Do use it in some CI environments that support it (Travis, CircleCI) That handles all the issues we've seen and everyone seems to be happy.

And more than that can cause compatibility issues. For example, its a bit of a pain to use utf-8 in the normal windows cmd.

Wasn't that issue fixed?

Really great advice here. But missing one key area:

Continuous Delivery / Change Management. I believe any policy without change management principles isn't really complete, especially when its about 12factor which is considered a gold standard for production.

* CLIs are notoriously difficult to update because you have to convince every single consumer to update it manually, otherwise you just have scattered logic everywhere. Having an update workflow is essential before releasing the first version in production.

* Closely tied, a clear Backwards compatibility policy.

Apart from those two major items, I have also found one optionally nice pattern to reason about CLIs:

Design CLIs like APIs wherever possible. Treat subcommands as paths, arguments as identifiers, and flags as query/post parameters. It's not always applicable, but doing that for large internal tools helps against the "kitchen sink" syndrome.

...all of these must show help.

  $ mycli

Some commands have an unambiguous meaning and don't need arguments. For example, it would be weird if a bare "make" command returned help information. Great post, though. I'm looking forward to digging into oclif.

In regards to 7, prompts are great for teaching new users how the program should be used.

Instead of failing then spitting out --help or manpage style info, the program just ask the user enter the needed argument or flag to continue. Having more ways to learn usage is always good IMO.

yep +1. A lot of times users are only ever going to run a command once. Better to ask for the right information than bailing out because it's not perfect syntax.

While a bit quirky, I really like powershells approach where you aren't limited to text streams. All the same advice applies.

Just the help factor alone is a big one.

I’m a huge admirer or well crafted cli apps, they can massively boost the effectiveness of a team.

An oldie but goodie here: https://eng.localytics.com/exploring-cli-best-practices/

> Follow XDG-spec Oh, please guys. Some apps even put visible (non .) folders in my home.

I have recently created single CLI program to put in all actions I need to automate. It's so fast to make new action, that I make anything that saves me just few seconds a day. Even stuff like "invoice" will open me timescheduling app, invoicing app and creates canned email to clipboard. "Clockout" will open timescheduling app and copy expected date and times to clipboard just to paste to app.

I've of course spent some time on automating Help, flags parsing etc, so I essentially just say what data I need and then what to perform.

It was best idea in a long time. I'm thinking that I'll publish framework for this as opensource (it's C# project).

> 8. Use tables

> By keeping each row to a single entry, you can do things like pipe to wc to get the count of lines, or grep to filter each line

> Allow output in csv or json.

Yes please. Default to readable-but-shellable tabular output, and support other formats.

libxo from the BSD world is a really smart idea - it provides an API that programs can use to emit data, with implementations for text, XML, JSON, and HTML:

http://juniper.github.io/libxo/libxo-manual.html

I personally love CSV output. Something like libxo means that CSV output could be added to every program in the system in one fell swoop.

>Still, you need to be able to fall back and know when to fall back to more basic behavior. If the user’s stdout isn’t connected to a tty (usually this means their piping to a file), then don’t display colors on stdout. (likewise with stderr)

GCC does this, leading to no colour output where it would be useful if you're building with Google's Ninja-build. Maybe there are some people who do pipe GCC output to a file - I've never had to. If you do this with your app, I'd appreciate being able to re-enable the colour.

Does these CLI features like tables, OS notifications etc work cross-platform (Linux, Windows, OSX)? IS there any good library to develop such CLI apps?

we use https://github.com/mikaelbr/node-notifier

> 1. Great help is essential

I like how this is their #1. In my opinion the best way to do this is with tldr.

https://github.com/tldr-pages/tldr

I'd highly recommend folks create a tldr page for their CLI app. Add 4-8 examples to cover 80%+ of the most common use cases. -h flags, readmes & man pages can cover the other 20%.

I almost want to rewrite the help section to encourage examples even more. They're incredibly valuable.

I hadn't considered this before you mentioned it, but oclif CLIs could integrate to tldr pretty well. It already supports arrays of strings for examples.

Yup. On CPAN, it is encouraged that the first part of your documentation after the table of contents is the synopsis[1]. The synopsis should clearly show how to do the common tricks with the library. From there you can link and refer to the more detailed documentation.

We're doing that for our internal CLI applications and it's great to be able to just copy-paste the common use case from the top of the documentation without searching much.

1: https://metacpan.org/pod/Carp

I feel the synopsis section of man pages often just becomes a bunch of useless garbage above the fold (for instance, look at `man git`).

Using it less as a complete docopts kind of thing and more of multiple common usages (like `man tar` and what you linked) is far more useful.

I think there is something here I hadn't really considered before. It's not an example, but also not a useless dump of flags. Food for thought I suppose.

Agreed on making examples more important. Powershell, for example, let's you "get-help command -examples" to just retrieve those.

Related: http://docopt.org/ – There are implementations for various languages. Whenever I need to write something that has a CLI, this is my default option…

Is it just me or every now and then Medium opens with what looks like to be a snapshot of the article instead of the HTML? I can't select text or scroll the page. If I refresh the page everything is normal, though.

I clicked this in hope that these was an article describing 12 CLI apps written in the Factor language. http://factorcode.org/

My hopes were crushed.

> I also suggest sending the version string as the User-Agent so you can debug server-side issues. (Assuming your CLI uses an API of some sort)

Isn't it some kind of disguised tracking? I know it doesn't give as much info as the user agent of a browser, but still, you could track the OS, even the linux distribution, and surely more, while still being a reproducible build.

Does someone has the name of the terminal app used in the article ? (with the nice colored path)

It looks like zsh with the Oh My Zsh [1] with the Agnoster theme [2].

1. https://ohmyz.sh/

2. https://github.com/robbyrussell/oh-my-zsh/wiki/Themes#agnost...

Thank you !

Nitpicks:

Replace “pipe content” with “redirect content”

nice catch, ty

"12 factor" anything seems to be a symptom of the over-complexity of modern apps, go back and rethink.

i found the original 12 factor website had a number of pretty reasonable suggestions based on experience of people who ran a business doing operations for other people's web apps.

sure, web apps are themselves probably over complicated, but given that you're doing a web app, the recommendations arent bad. compare to where things have gone since, with containerisation.

In a lot of ways the "12 factors" have overly simplistic views of the world. For instance, storing your config in the environment is fantastic -- until you remember a great many frameworks will dump their environment to the browser in a number of failure scenarios. There go all your secrets.

Reply on news.ycombinator.com