Hacker News Reader

Comment by ralph | Hacker News Reader

This is very bad advice. I won't thank you for making me check GNU grep's --ignore-case is precisely the same as the very well-known, POSIX'd, -i. And ditto for all the other verbiage that clutter and obfuscate the script's intent.

Short options should be used where they're the more typically known and standardised. Long options are for the unusual.

    grep -iv
    sed -n
    tr -dc
    awk -f
    ls -tr
    comm -23
    tail -n

View on HN

I'm a 4th year student in a well-respected CS program, I have used linux off and on for over three years now, and I would consider myself perfectly capable of maintaining a shell script using any of the utilities you mention. However, without checking the man pages I could not tell you the behavior of the -n flag for sed, the -dc flags for tr, the -f flag for awk, or the -tr flags for ls, and I only have a guess as to what comm -23 does.

This is not the bread and butter of programming, where everyone worth a damn should be able to decipher "int i = 10;" even if it's not their brand of syntax. There are people who are perfectly capable of understanding and maintaining a piece of software that uses "tr --delete --complement" but would be dumbfounded when confronted with "tr -dc" until they looked it up.

> This is not the bread and butter of programming...

This is, however, the bread and butter of shell programming.

> ... "tr --delete --complement"...

And, likewise, I'm not sure of the meaning of --delete and --complement, but I do understand "tr -dc". I'd have to look up --delete and --complement to work out what they're the equivalent of.

These days I think it's clear that the dominant use case for shell scripts is not a full time responsibility but rather edge work for deploying/installing code written in other languages. Most people looking at shell code aren't going to be shell masters, they're going to be engineers of more modern things that have to look at the shell script for some reason.

> ...they're going to be engineers of more modern things...

Have you read TAOUP? The entire Unix philosophy hinges around the shell, since the shell is the tool that lets you glue different pieces together. The shell is not outdated.

If you live entirely within one language ecosystem, then you may be tempted to consider the shell an afterthought, and make all services available via API calls instead. But this is inefficient, slows down development, reduces flexibility and limits the capabilities of your programs. This fallacy is old and certainly is not "modern". See TAOUP for details; it explains this far better than I ever could.

But a lot of the defaults in the bourne shell and various posix utilities are wrong. One example: space as a field separator is clearly the wrong default, since it's nearly impossible to not find file-names with spaces in them these days.

"This is, however, the bread and butter of shell programming."

This is really old-fashioned thinking. The people who have been in this industry for 10+ years all grew up without computers - and thus had to memorize all of the flags, obscure shell commands, and weird regular expression. What was taught in school back then was to do things perfectly the first time, because CPU cycles were expensive, and bugs were time consuming.

The generation that graduated from college about 2 years ago is the first one that had universal access to a search engine since they knew how to use a computer. This generation (and all future ones for the rest of time) sees memorizing things like this as more of a waste of mental space. What is taught in Universities now is problem solving, research, and larger concepts. There is very little straight memorization, and almost zero programming by hand/whiteboard programming. No current college grad will be able to write a shell script without reference material, but every college grab should be able to write any simple script in less than 5 minutes with Google.

They probably think they invented sex too. ;-)

Memorisation isn't the issue. Knowing the tools available and their options is. That comes with study. Unix command line is a boon for problem solving and experimentation, just go and read Jon Bentley's _Programming Pearls_. One can't Google for what one doesn't realise can be done.

You're going to have a hard time making it through Hemingway if you have to reach for a dictionary every time you come across a word like "middleweight" or "khaki", even if your dictionary is online and answers your query in less than a second. You need to memorize the meanings of a vocabulary of words in order to read fluently or write well. That's not "old-fashioned thinking". It's just common sense. And it's just as true of shell scripts as it is of electrical engineering, mechanical engineering, poetry, or math.

While both the above words are known to me, I do like that when reading a book in Kindle on my iPad, I can tap a word, and have its definition show.

Is that a signed integer or unsigned? How many bits? What's the lifetime of i? One needs to know more than is inferred by a few characters. Seeing the long options just makes you think you understand what it's doing without knowing what it does. The latter comes from study; there's no short-cut. Fortunately, Unix is worthy of study unlike for example the Windows API.

I can't help but agree with this sentiment. Generally, if the operator doesn't understand what a command line option does and isn't willing to read the man page, they're open to a whole slew of unpleasant surprises (although I think this applies to many things, not just userland utilities). Long options simply provide a false sense of security and may not always do what someone thinks they're supposed to do. Worse, they may not always be available as others have pointed out insofar as the BSD flavors of specific tools provide.

I like what you said about "study." There truly is no shortcut for reading documentation available with the system, and assuming by inference what a command is supposed to do based on its options when neither the command nor options are understood just seems to be horribly, horribly bad practice in my mind.

Unless I am doing code review, I don't read code with the intent to find latent bugs or insecure idioms. I assume the programmer who wrote the code was competent, and wrote a correctly functioning program. (If I cannot assume this, then why am I using the code at all?)

If I am doing code review, then I have the man pages open anyways. If I am trying to write bug-free code myself, then I have the man pages open anyways. In these cases it doesn't much matter what form of flag I use. But the most common case of reading code is a brief scan trying to grok what the code does, which shouldn't require frequent reference to man-pages in much the same way that reading a novel shouldn't require frequent reference to a dictionary.

I understand your concerns about false sense of security and unexpected behavior in corner cases. I've browsed the IOCCC, I've browsed the CVE database, I know how easy it is to hide nasty behavior in unexpected corner case interactions, and that the only defense against them is vigilant attention to documented behavior.

To my mind the most common interactions with code, in decreasing order, are as follows:

1. Executing it. (Flag agnostic, portability issues aside.)

2. Reading it. (Descriptive flags >>> Cryptic flags.)

3. Maintaining it. (Standard flags >> Obscure flags.)

4. Writing it. (Short flags > Verbose flags.)

So I would say the most important thing is that flags are descriptive of their behavior. The second most important thing is that they are common and standardized, but this isn't as important as descriptiveness -- a less common flag that describes its operation better wins. The least important thing is brevity, it only matters once.

> then why am I using the code at all

You're using it because it comes with the job that's paying you and you want to see what it does and form an opinion on its quality. Much crap code is procured and produced by companies.

“…clarity is often achieved through brevity” — Kernighan and Pike, _The Practice of Programming_. It's an excellent book, I recommend every programmer should have read it. http://amazon.com/exec/obidos/ASIN/020161586X/mqq-20 http://cm.bell-labs.com/cm/cs/tpop/

If you aren't sure what the "-<x>" flag does for a particular command, but you are a competent Unix user, you should be fine with the man page or 10 seconds of Google. If you aren't used to Unix, then I think it's dangerous to expect the verbose flags to explain things simply to you. For example, what would --mmap or --null-data mean to you in the context of the grep command? Are they immediately obvious?

The difference is that with long flags, even someone without 10 years of POSIX experience knows what's going on. The long example on the OP's article makes a lot of sense to me, even though I'm primarily a Windows user.

At the same time, for instance, I've no idea what the -tr flag to ls does, even though I use ls often enough.

You are not a Unix user. You don't know Unix's commands and their common options. If the article is aimed at you and people like you that read your scripts then it's fine but it's akin to commenting "dir /w" in a DOS batch file to explain its purpose. Unix users should not following the article's advice, they should be embracing Unix's style and ethos; it's part of what's made it such a success.

BTW, a month of using Unix should have a Unix user knowing what ls's -t and -r flags do, along with -a and -l they're some of its most commonly used.

I'm sorry, but this is terribly pedantic. The world isn't divided between deep experts of a technology and everyone else. Case in point: me, a moderately tech-savvy person. My main computer is a Mac and I'm a casual user of bash. I struggle to get things done every time I have to do something more complicated than "show me the files of this folder, even the hidden ones, even in the subfolders, then dump them in a .txt". The obscurity of flags don't help.

I completely agree. I use various flavors of Unix every day at work, and I can only identify half of the flags mentioned at the root comment.

This reminds me of the whole "replace" debacle [1] a couple weeks back, where the power nerds jumped all someone who wrote a bit of code to simplify common tasks. It's hard to understand how some people think flexing their arcane knowledge in everyone's face makes them look good. They trash efforts to make software more accessible and maintainable so they can pretend to be king of a tiny hill.

[1] http://news.ycombinator.com/item?id=5106767

Then you don't use Unix, you merely enter a few canned commands to perform a limited set of actions. You are not making it work for you.

Well, but maybe that's the point of the original post - it is talking about maintenance of shell scripts; so that implies the following:

1) It is not a command that someone "uses", it's a script written by a programmer to be run for a business task or an end user quite likely may not "use Unix" (by your definition) ever.

2) The script goal most likely is not to make Unix work for you - it was originally written to make Unix do task X for you; but the goal/question in the OP is how to make that program more maintainable. Maintainability is an important goal, and to achieve that it's definitely acceptable to mutilate the way you usually do one-off tasks manualy.

3) The shell script is written for non-shell-script users - it's quite likely some glue for a system where 99.9% is in other languages, and the maintenance guys will specialize in those languages, and may not "use Unix" in their daily tasks at all. It's very common and reasonable to write all your code on a Win or Mac computer and then have it deployed to some linux server; and most companies currently do have separate ops teams (which would "use Unix" every day) and developer teams, which might "use Unix" once a month or less, even if the end binaries run on Unix.

The OP says to use long-options at all times when not at the command prompt. All this other stuff is a scenario you're placing around it to try and justify it.

I'm all for being nice, but I think you are missing the point. It's not about flexing knowledge for ego's sake, that's entirely beside the point.

No, the source of Unix nerd's retort is the realization that a small set of simple, standardized and very sharp tools performs better over time than an ever-increasing set of intuitive tools to suit a narrow use case. This is not to disparage anyone for writing something like that replace project. You certainly don't deserve scorn for that, and if that tool finds a healthy place in your utility belt then all the better.

However if you do a lot of work with text files on unix then I think inevitably you will reach a tipping point where it is in fact easier to learn and remember a limited set of commands and flags than it is to remember the ideal mountain of simpler commands that do all the things you regularly do in a more beginner-friendly fashion. If you want to argue against "arcane" unix tools being better you have to confront this argument head-on without assuming the proponent is an insecure, anti-social neckbeard out to prove his own superiority.

> a small set of simple, standardized and very sharp tools

This may have been true(r) twenty or thirty years ago, but in a contemporary Unix installation, it is not true at all.

The set of tools is often NOT small (there are thousands of them); they are NOT simple (two examples: 1. the ls man page and the huge number of options it takes; 2. shell quoting rules); they are NOT standardized (I regularly run into incompatibilities between BSD-derived and GNU tools, for example, the -E option for grep); and they are NOT sharp.

In summary, the whole "small tools that one thing only, but that do that one thing well" thing is a stupid, outdated mantra that may have made more sense many years ago, but not anymore. For some reason, people keep on blindly believing it even though it bears no relation to the reality of modern Unix programming.

I would respectfully disagree here. I can't remember, and won't bother to look up, how to recursively use cp / mv / scp / rsync / whatever else... but I'd bet it's -r, or -R. I'm not positive how to tell a shell program "I REALLY MEAN IT!", but I''d guess it's -f (unless it's kill, and I'm not going to sympathize with gripers over that one odd case :) ). There really is an (admittedly idiomatic, but still present) intuitive nature to these things. There are loads of counterexamples, for sure. But still - say I want to use extended Regexp in grep, or sed. I bet it's "-e" or "-E", right? Plus you can always search the man-page, without bothering to read it all.

Reading man-pages does do a lot for you. And in my experience, I don't spend a lot of time while on the same engagement/project/whatever writing shell code across BSD/solaris/GNU-linux/etc.

And honestly, if you can write a sophisticated sed substitution for BSD without reading the man page, is there really THAT big of a barrier lying between you and writing the same thing in GNU/Linux, or non-POSIX-compliant sed, or whatever else? You read the man page for 10 seconds and find the flag that means what you want to use. If it's different, you explore.

This is totally not a rant at you, and I sincerely hope you don't take it that way - I just haven't found myself utterly foiled by the differences between say, RHEL v SUSE v BSD recently. There are differences, but hell - there are differences between versions of languages, between terminal emulators (scripts that work in bash but not in zsh and vice versa), etc.

Sure unix has grown more complex over time, but what's the alternative? Start from scratch with simple commands? By the time you approach a system of equal power you'll have hundreds of thousands of commands instead of just thousands. For all its warts, it's hard to build a better shell than what's available in unix.

Sure; I'll elaborate. First, I am arguing against the unapologetic championing of arcane knowledge, not the tools. I agree that awk/sed/grep/etc are powerful, and knowing how and when to use them is highly useful.

I take issue with the attitude that the way these tools are used, with their billion cryptic parameters, is some flawless pinnacle of achievement.

Some people have put a lot of effort into memorizing Unix switches already--but the difficulty doesn't mean that work was meaningful. Unfortunately, to justify their sunk cost, a subset of those individuals tout their knowledge as something by which the rest of us should be impressed. In fact, this comment thread is littered with disdain for peers who dare to tread on their domain with "human-readable names" and "documentation".

In the root article, the author simply suggests that it is more maintainable to use full names for arguments because the code is then self-documenting. It's like using completeVariableNames instead of sht_y1s. Actual comments would suffice, but most code (unfortunately) isn't commented or the comments are done badly.

If a programmer is solo, whatever. Go nuts. Use shortcuts; don't comment. Hard-code paths in your scripts and write 600-line functions. A coder only has herself to blame for the problems she caused.

However, when programming with a group there will always be coworkers that can start being productive sooner (without interrupting you) if things are named and described in plain English and do exactly what it looks like they do.

Script switches are not a remotely interesting problem. Which style you use is a meaningless debate--until you factor in time. Time to debug. Time to look up documentation. Time to memorize. Time is a truly scarce resource, and there exists a better way than rote memorization: making code work in an obvious way. Actively choosing not to employ this idea steals time from others and those of us that lose minutes so a shortcutting programmer can save seconds of typing don't appreciate it.

That said, it's up to one's discretion about what "obvious" means. But nobody should try to argue that making things more readable is inherently bad.

You have a point about the -t and -r flags, but,

> Unix users should not following the article's advice, they should be embracing Unix's style and ethos; it's part of what's made it such a success.

I always wonder why this is taken so religiously? You're not the first person to write a comment like this, so please don't take it too personally. But is it not possible that the Unix style and ethos is mostly great, but there's a few things here and there that could've been better? You seem to imply that a culture that prefers short options is one of the reasons that Unix became a success. I'm not so sure.

In fact, if `dir /w` had a long version of the option, i would've indeed used it. I think it's a shame that most native Windows commands, unlike their Unix counterparts, don't have long versions of switches. But I doubt that that's contributed much to Windows' success.

Unix is in no way perfect, for that look to Plan 9. ;-) But just as one should try and understand the idioms and style of a programming language rather than write Pascal in it regardless, so one should try and embrace Unix then suffer and learn from its warts rather than write VMS DCL in it.

And yes, brevity is pretty fundamental to its early culture. Unix was created by mathematicians and scientists that were using it every work day to get stuff down and they recognised and extolled in their writings the power of notation. Imagine maths without superscripts, Σ, etc. So it is with Unix. More wordy notations existed in other OSes at the time, and more noisy commands, e.g. VMS would tell you that "dir/size/owner/prot foo." completed normally, even though that's hopefully the norm. It gets very annoying. :-)

Unix was developed on a system with a 110 baud teletype. On such a system, brevity was the path to sanity (seriously, imagine programming on a manual typewriter that only supported 10 characters per second).

VMS was written later, with a faster interface. Also, you didn't need to specify the entire long option, you only needed to type enough to disambiguate between "/all" and "/almostall" (for instance).

The constraints of the time shouldn't be confused with their recognition of the power of notation.

Also, based on the poor advice of this thread's subject, surely one should always write /almostall in one's DCL script for clarity and to lessen the chance of ambiguity as the command evolves? grep's -i is only ever going to mean -i.

VMS died, DCL was often ridiculed, its over-verbose style was a part of that. Its lack of power played a larger part.

Is Randall Munroe a Unix user after 15 years of using it?

http://xkcd.com/1168/

That's a comic. He sells a shirt on his site's store that is filled with Unix tips, including "tar -xf # extract anything". I'm not sure this is a legitimate criticism.

Unix is an operation system, not a religion. I would argue that the pragmatism and adaptability contributed more to its success than its style.

There's something to be said for the automatic recognition that something like "tar zxvf foo.tar.gz" has, versus the verbose "tar --gzip --extract --verbose --file foo.tar.gz." It feels terribly ponderous to do the latter. That said, when I see a command I don't recognize immediately ('sed -n' comes to mind for some reason) I would benefit from seeing the long option. The other side of the coin is that once I go look up 'sed -n' I'll probably remember it for next time.

http://xkcd.com/1168/

Aside,

    sed -n /re/p

is grep. sed's default is to print every line at the end of the script, -n says not to. Larry Wall rightly had perl(1) inherit -n along with perl's -p.

I always remember this as "eXtract Ze Vucking File".

(You have to say it in a bad german accent.)

We had a very similar discussion over the weekend at http://news.ycombinator.com/item?id=5157215 but with the domain expertise being with non-programmers.

My opinion remains the same as it was there. If you've already gone through the effort of learning the short options, there is no reason not to continue amortizing that effort by making use of them. On the other hand if you have not really learned the short options, it is only worthwhile doing so if you plan to use them enough in the future that it will pay off.

Of course if you are writing code that you expect to be maintained by someone else, aim it at the fluency that you expect to be able to demand at them. Aiming it lower than that will just make things more painful for you. Aiming it higher will make things more painful for them (and the vast majority of the cost of software development is in maintenance, so they matter more than you do in the long run).

And at least a few of these (grep -iv, sed -n, ls -tr) aren't even unusual. If you ask what -iv means in grep, I assume you have used grep < 10 times ever.

Reply on news.ycombinator.com