Hacker News Reader

Comment by dap | Hacker News Reader

IMO, this is an anti-pattern. It's violates the principle of least surprise. (How come I see X when I run the command, but I can't grep for X in its output? How come it works when I run it from my interactive shell, but it's broken when I run it from a script? And things like that.)

View on HN

Indeed, the GNU Coding Standards explicitly argues against doing that:

“please don’t make the behavior of a command-line program depend on the type of output device it gets as standard output or standard input.”¹

① https://www.gnu.org/prep/standards/standards.html#User-Inter...

Interesting. Doesn't git (a pretty new tool) violate this? I think running log from the terminal pauses per page of output but if you pipe it to something it pipes all the content.

I think it's just piping it to another program. It's still giving you the same output, but sending you to a program meant for humans to be able to read text in a console, instead of just printing it all out.

That said, I'm not entirely sure which git pipes to.

I think it depends what sort of things you use it for. I often use it to switch on or off ANSI colourization, which doesn't really violate the principle of least surprise.

When used sparingly and thoughtfully, I've never personally had an issue with it.

Good point, and it makes me even more convinced that all rules of thumb have important exceptions. In fact, I've also used tools that use ANSI colorization, and disable that when not talking to a tty, and wished that they wouldn't because I was piping them to "less -R" :) (At least that's a graceful failure mode, and it's pretty clear what probably happened even if one doesn't exactly understand the details.)

I've always thought it would be nice to have a utility like cat that I could pipe commands to, which would trick them into thinking all their streams were attached to a tty, so you could do "uses_colours | ttycat | less -R".

I'm sure it's possible, but you'd have to acquire a new pty and decide what termios settings you want. It's a nontrivial hack, I think.

I'm actually kind of surprised it's not in moreutils[1].

[1]: https://joeyh.name/code/moreutils/

EDIT: Hmm, maybe it's not possible. I can't figure out exactly how to do it, anyway.

EDIT again: Apparently `script` does this on Linux: http://monosnap.com/image/Qlig4CHmQgV9pxvSmndVUgMTU88Adz

EDIT again: Expect's `unbuffer` also does it: http://expect.sourceforge.net/example/unbuffer.man.html#toc

It’s not possible in the way you think, because the two programs don’t know about each other. All the pipe character does is put the stdout into the stdin of the other program.

So this feature must actually be present in the shell and maybe it is. I’m no expert but maybe zsh already offers something like this?

> I've always thought it would be nice to have a utility like cat that I could pipe commands to, which would trick them into thinking all their streams were attached to a tty,

Daniel J. Bernstein wrote a "pty" package back around 1991 that did this. Version 4 of the package was published in 1992 to comp.sources.unix (volume 25 issues 127 to 135). It's still locatable on the World Wide Web.

Bernstein later updated this, around 1999, with a "ptyget" package that was more modular and that had the session management commands moved out of the toolset to elsewhere. The command from that package to do exactly what you describe is "ptybandage". There is also "ptyrun". Paul Jarc still publishes a fixed version of ptyget (that attempts to deal with the operating-system-specific pseudo-terminal device ioctls in the original) at http://code.dogmap.org./ptyget/ .

As a bonus feature for people who use source code, there are similar "ptybandage" and "ptyrun" scripts, for which you will need Laurent Bercot's execline tool (http://skarnet.org./software/execline/), in the source archive for the nosh package at http://homepage.ntlworld.com./jonathan.deboynepollard/Softwa... . These make use of the terminal-management tools in the nosh toolset.

With both of these, you should be able to run "ptybandage uses_colours_if_tty | less -R"

You may not have issue with the sort of things you use it for, but others might.

For example, I run shells in Emacs and have had to tweak loads of shell scripts written by colleagues to fix their poorly-implemented colourisation. It's useful to know when a test has failed; it's not so useful to have the whole terminal set to white text on a pale pink background.

One day I couldn't SSH into our servers from Emacs. It turned out somebody had edited .bashrc for the admin user to make the bash prompt blue. Emacs' TRAMP process was looking for a prompt ending in "$" or "#", not "$\[\033[0m\]", so it didn't realise the connections were successful.

There are two ways of handling this: we can blame the source of the bug (the person adding the colours incorrectly, or the assumption-loaded TRAMP regex), but there will always be more bugs in situations we'd never think of. Alternatively, we can avoid being 'too clever', and instead aim for consistency and least surprise.

Are you suggesting that colored prompts violate the rules of consistency and least surprise?

(Actually, if you are suggesting that, I'm not going to disagree. But I am going to say that if so, those rules don't apply in the case of colored prompts, because colored prompts are useful.)

I suppose I'm suggesting that, aside from personal scripts, we shouldn't assume too much about who our users are and what they're trying to do. The principle of least power tells us to use the dumbest format that will work, eg. plain text.

Anything we add on top of that, eg. ANSI colour codes, will be useful to some but harmful to others. The tricky part is working out which of those categories the current user is in.

So is your proposed solution not to have colored prompts? (I vehemently disagree.) Or not to put them in a .bashrc? (I still disagree, but only strongly.) Or something else?

The main problem here is that a change that should have been done as part of an individual's customization was made to a shared configuration file.

Or, y'know, have coloured prompts but place the escape characters such that they don't mess with prompt-detection regexps.

To be precise, what you're suggesting is that we have prompts which are allowed to be colored except for the $/# at the end, because you can't color those without following them by escape characters. And that prompts must have a $/# at the end.

I don't consider that an acceptable solution.

As much as I love Emacs, that sounds like it's a bug to assume that one's prompt will follow a convention (ends-in-$). The convention is useful and good, but it seems strange to blame someone for breaking your tool's expectations when they had made a valid prompt.

I agree, especially for the behaviour from the parent:

"ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live

How people usually discover what these commands do is by running them interactively, and if that results in some output being hidden vs being run noninteractively, then they have little reason to believe that it could yield more output than what they're used to seeing. I think a certain number of "ps" users don't know it can display full paths and commands, if they've only ever used it interactively.

On a quick look through ps's manpage I couldn't find anything about this. Am I missing something?

-w

It doesn't say that it'll do that automatically when run through a pipe.

Yep, and the result is I have needless sprinklings of "www" in my shell scripts cause of habit. Technically I don't need it the moment I pipe into grep, but oh well ;) Anyway I personally dislike the SysV-like 'I need to add stuff like -aef' to get all that on-the-screen/into-grep vs the BSD-like it knows about TTY but I can convince it otherwise into stuf like 'less -S' - personal taste I guess.

Actually I just found this in the 'ps' manual, it looks like the output width is actually undefined! "If ps can not determine display width, as when output is redirected (piped) into a file or another command, the output width is undefined (it may be 80, unlimited, determined by the TERM variable, and so on)."

Ideed. I've seen people over and over stumbling over this weird behaviour.

It may have some merits, but as a general advice this is definitely an anti-pattern.

Another example is "curl", where "curl URL >outfile" is chatty on stderr, while "curl URL" is quiet on stderr. That's very annoying for scripting, you easily forget to set "-s" in your scripts due to that behaviour.

And yet .. programs are not just for composition. They have to behave sensibly for people.

I love that 'git log' outputs in a pager. 'svn log' by comparison is nuts.

That's an interesting case. Should it not just output to stdout and you have to pipe it to less? If you wanted it to always pipe to less you could just set up an alias for it.

The git diff and log commands are primarily intended for human consumption. I'm not the one you were relying to, but I think it makes sense that git defaults to make that consumption easier, even if it isn't strictly "Unix". (You can also disable this behavior and get a more Unix-y interface by default via gitconfig or on a case by case basis with --no-pager).

For sure, and in fact, I very happy with the current behaviour. I guess the pager is a case where it doesn't matter much if it's included or not since I guess it doesn't get in the way if you pipe to something else. Does it consume extra resources?

    git log | wc

    git log --no-pager | wc

I'm sure it's neither here nor there in practice. More of a hypothetical question.

I've just realised that the best solution to this (which I've never seen) is for the shell or terminal emulator to capture long output into a pager for you once it exceeds a certain length.

I think there may be some flags in pagers to do this for you, but I can't remember for sure.

I believe software should be written to to satisfy the common case and the common case is the non-expert.

I like the default pager in 'git log', too.

Some work has been done on the svn side: http://svn.apache.org/repos/asf/subversion/branches/automati...

However, it's not on trunk yet because it's hard to find good defaults. An automatic pager makes sense for some commands, but not all -- and in a meritocratic development model this kind of thing can cause an endless discussion... I suspect we'll eventually merge the feature in a disabled by default state and allow users to enable it on a per-command basis.

Git's hard-coding of options passed to the pager has problems, too: https://mail-archives.apache.org/mod_mbox/subversion-dev/201...

This behavior of Unix programs is basically the same concept as Perl's "context" (list vs scalar), but even moreso.

Yea most commands don't do this though. Most commonly used for colouring output.

ls is a bit more than just a command though. It's part of the furniture and prehistoric.

I have run into this problem when trying to automate certain tasks on UNIX boxes.

Dealing with programs that act differently depending on their output device is very annoying.

I despise it when commands do this - mysql -e results are formatted differently depending on whether the output is directed to the terminal or to a file.

Reply on news.ycombinator.com