The hard part in becoming a command line wizard
johndcook.com
johndcook.com
If you've memorized the hundreds of different invocation arguments for find, sed, grep, tr, sort, uniq, cat, xargs, etc. -- then yes, you can be very fast. If not -- be prepared to RTFM for about 4 different programs to get your job done.
That's an incredible hurdle and it really only makes sense if you are using these tools daily/weekly. I'm still not completely convinced that its "worth it" to master these tools, when general-purpose scripting languages are almost as succinct and obviously way more powerful. That's speaking as someone who's written a fair amount of Bash.
In contrast, I think tools with higher discoverability and self-documenting nature (arguably, Excel) usually offer a more user-friendly way of doing the same thing, unless you are doing it a lot.
Furthermore, it's often quite easy to slightly tweak a command I found online, even if I don't understand much of it.
I work in visual effects. Most of our software has a build-in Python interpreter. I was talking to a tech savvy junior who was interested in becoming a full time modeler/artist. He was asking if he should learn programming/Python and I advised against it (unless it scratched his particular itch). I felt his efforts were best put towards getting better at modeling. Most large places I've worked at the artist can get by with a sticky note of common commands. There's usually someone they can ask for advice or small scripts when things get hairy. Smaller places you're more on your own. But it is surprising when you have 100+ items, you need to be sure they all have 2 sets of UVs. The manual way is to click on each one and pull a drop-down menu or with Python you can write a one-line foreach to test and print. It's all about knowing which screw to turn[1]
I often challenge myself in doing something in a single line just to scratch my own itch. Often when tackling daunting tasks (do this thing 20-50x) I weigh the benefit of doing it manually, but bias a tad toward automating even if I'm throwing the code away. But I don't think it's worth most people's time.
The biggest benefit of working the same place a long time is knowing who to ask about things. Second is knowing the history of things. Third is having a junk drawer of scripts that don't quite work out of the box, but they have idioms I can grab when I need to hammer something out.
[1] https://calvincorreli.com/blog/1397-knowing-which-screw-to-t...
I'm also a big believer in mastering your tools. An artist is limited by their imagination - and imagination is largely structured by your knowledge of materials and the capabilities of tools. If you're working with 3d models, and you plan on mastering it, having an idea of what you can do with An artist is limited by their imagination - and imagination is largely structured by your knowledge of materials and the capabilities of tools. It spurs you on to try new and ambitious stuff, new styles, new forms.
My career has let me focus on the backend and internals. I know people way more proficient than I am with Maya, but they know little to nothing about Python or the DG (or maybe MEL). I know a few vfx sups who can cobble together a basic shell command, but aren't masters and don't know a bit of Python. It's much easier to interact with them (especially over chat/email--I can send them a command and trust them to modify paths and run it), but it's not worth their time to invest in honing those things. It's probably cheaper to fly someone from a different state to do these things for them than it is for them to spend half a day writing something to parse and edl to rename some Quicktime files or troubleshoot a bad power supply in their workstation. I've also noticed that the more technical they are, the deeper the hole they're in before they ask for help. Hopefully that means they're more productive (and are more ambitious), but it usually means I have to ask them what their original problem was, then what was the first thing they did, then the second, and unwind the situation.
For the situation you described of turning buildings there are probably 5-10 completely different approaches you could take (apart from manually placing them), only a few use Python. At a moderately sized studio (on a moderately sized project), I'd reach for something like Houdini to do what you're asking and you would be hard-pressed to find someone who would use Python for this kind of thing in Houdini. On the flip side I was handed a project at a small place that was well suited for Houdini. We didn't have a license so I cobbled something together in Maya. The producers were telling me in their experience they'd hire someone who would do a project in Houdini, but later when they needed changes and that person wasn't available they'd have to hire someone to redo it in Maya. Reality is hard.
My thoughts about this really clicked with photography. A lot of photographers are just technical enough. Most of the process is a black box, but they've had their camera for years, tested enough inputs and outputs that they can trust their tools to do what they want. Other photographers dig into the tech. They can't, by definition, have as much time in the field. They can tell you a lot more "why" and might be equally good or better in many respects, but it can be really easy to get lost in the tech and miss out on the practice.
On the one hand, you're absolutely right when you say that it's largely unproductive to work outside of your speciality. On the other, I think it's a bit like painters learning to make their own frames and mix their own paints. It's not strictly necessary, but it deepens the understanding, and it opens the door to loads of stylistic discoveries.
I think the thing I like about python, or programming in general as a creative technique, is the combination of amazing fine-grain control, with absurd expressiveness. The kind of techniques you learn as an artist are usually strong on one or the other. Crosshatch gives you a lot of control over light and shade, but it's a time-consuming, boilerplate kinda technique. Most traditional techniques are like that, or, like throwing paint at a wall - very expressive, but no control.
Programming the creation of sculptures and images is the kind of amazing opportunity that it feels sort of crazy to pass up. I've spent an inordinate amount of time refining my ability to draw nice gradients with a set of pencils - and now I have a machine that can give me a perfect gradient with a couple of lines of code!
I like to phrase it as "Linux assumes you know what you're doing. Windows assumes you don't know what you're doing. Mac assumes you don't want to know what you're doing." Working in an interface that doesn't hold your hand can be liberating when you understand how that system works, but there must be a way we can step back.
Maybe it would be worth adding something similar to the drop-down auto-completion boxes I see in so many modern IDEs. That way instead of reading the man page of every single command and program, I can (for example) get a list of possible arguments, and each of those arguments provides a summary of its function when highlighted. This could all be done in text, so that it would be available outside of the window system. Maybe it has already been done.
Obviously such a system would be laborious to implement and obnoxious to seasoned bash users (though it should be easy to disable), but it would be extremely handy to those who (like me) only occasionally use Linux. In any case, something should be done that makes the system a little more welcoming to casual users.
But I get what you're saying - even when looking at the PowerShell design, script documentation is a first class citizen there. As a scripting language I'm pretty impressed with it, but as a shell I find it an annoying place to be in. OSX on the other hand simply has a unix shell, with a respectable amount of CLI tools available to interact with the desktop. Sadly, Apple scripting has been a bit neglected in the last few releases and is not really supported anymore by all applications.
disclaimer: I do not like Linux on desktop. My daily driver is a MBP with OSX, and I used to work on Windows (but still have a gaming box), and all my linux boxes are servers.
Couldn't agree more. The syntax is fantastic for scripting as it has good support for objects, but it is not good as a shell due to the different return types and polymorphic behavior of some functions based on the type they are operating on. In *nix land this isn't a problem as everything is text.
What made a huge difference for me was setting up Arch Linux on my XPS 15 laptop. Following the install guide[1] forced me to go through the process of standing up a functioning Linux desktop from a much lower level starting point than I'd ever done with Ubuntu or CentOS or even Slackware. The whole experience reminded me of the first few times I built PCs as a kid: At first they were intimidating boxes full of complex circuit boards and imposing nests of ribbon cable, but putting a few together demystified all that in a hurry. Setting up my Arch laptop had the same result as it relates to Linux configuration in general and command line operations in particular.
[1](https://wiki.archlinux.org/index.php/Installation_guide)
The reason s-expressions are better than text is that they are more expressive. They allow arbitrary levels of nesting whereas text doesn't. Text naturally develops a record structure with only two levels of hierarchy: records separated by newlines, containing fields separated by some other delimiter (usually spaces, tabs or commas, sometimes pipes, rarely anything else). Going any deeper than that is not "native" for text. That's why, say, parsing HTML is "hard" for the unix mindset. But a Lisper naturally sees HTML (and XML and JSON and, well, just about everything) as just an S-expression with a more cumbersome syntax.
That has got to be a joke right? S-expressions are a subset of text it’s axiomatically impossible for them to be more expressive.
However, what would you expect from someone whose username is literally "lisper" ?
By using specific constraints (rules for sexprs), one creates a wider array of possibilities.
By having no rules (text), one has to provide a rubric on how to parse the text. And when you have to parse, you have to write a parser, and define the parsed format.
This reasoning is backwards. Structure creates expressiveness. A novel in correct, readable English is more expressive than an arbitrary pile of letters, because it follows a form. A painting, even when represented as a JPEG, is more expressive than a large number, because it is considered as a valid JPEG and a painting. A Bach chorale with no parallel fifths, and a mid-20th-century experimental atonal piece, are both more expressive than banging on a piano.
The Unix standard of lines within which there are fields delimited by some character is, itself, a very primitively constrained/structured form of text, so it too is less expressive than raw text could be. But this is a daft discussion.
So, now we have two subsets of text, and it makes sense they can have different levels of expressiveness.
(Actually, unstructured is too extreme since there is a little structure. Something like "isomorphic to CSV" isn't quite right, but closer.)
The goal of the definition isn't to give S-expressions an advantage so they can be proven superior and Lisp people can win the argument. It's to describe the sorts of files that actual existing command-line utilities (like this in Unix/Linux) can conveniently operate on.
The "head" command deals in line-oriented fashion and can print the first N lines of a file. The "cut" command can select certain columns based on a line-oriented and character-delimited structure. There is no exact definition, but this gives you a general idea of what text files means in the context of "becoming a command line wizard".
s-expressions are structured text. Therefore, for some set of problems, s-expressions are "better" (contain more information, are more easily manipulated, etc.) than unstructured text.
I mean, I get it. Lispers like lisp, and think it’s “better”. But it’s more more expressive than text, and in fact operating on text gives you natural access not just to s-expressions but to any other programming languages or structured and unstructed examples of text.
If you shell operated only on s-expressions that would either have you reduce expresiveness, or just doing tricks to encode the text-stream as a s-expression which hardly qualifies as making it more expressive, that’s just adding extra redundant information around the original information.
Similarly, a C compiler operates on well-formed C programs.
Neither one can do its work on arbitrary text or on the other's text.
You can argue whether one is better than the other, but while each is a subset of "text" the limitations of the subset enable the expressiveness.
"I work on matter" vs "I sculpt" or "I paint" or "I cook".
We're talking about a class of strings, not a single individual string. S-expressions are more constrained than arbitrary text, and therefore contain more structure than arbitrary text. Constraints are information.
You can't simply impose a tree structure on any arbitrary piece of text.
The only popular modern incarnation of Lisp philosophy in shells is... PowerShell - where instead of s-expressions, the tools exchange objects. That's AFAIK binary blobs, but if you wanted to retain a text-serializable form, then the most obvious way to do it would be... S-expressions.
I mean, there's a reason why regular expressions were one of the first things that went into Unix. But with lisp, you need neither regular expressions nor separate parsers for each format, because s-expressions enforce a uniform encoding of all structure in a data structure (a tree), which is parsed automatically by the interpreter!
Edit: And, as lisper clarifies, "S-expressions are more expressive than text that adheres to the typical unix conventions" [0], meaning that s-expressions make it trivial to arbitrarily extend that structure (whereas doing the same beginning from Unix conventions gets messy very quickly; see, e.g., lisper's example of parsing HTML).
First of all, the notion that a subset being "more expressive" than its superset is "axiomatically impossible" is wrong. If I form a 20 letter string that's a sentence (e.g "Welcome to my house", it would be more expressive than the totality of 20 letter strings (even if it's a subset of them). That's because the totality of 20 letter strings includes all possible valid sentences and close to 20^26 (or more if we add space, numbers etc) arbitrary BS strings, and thus can't really express anything concrete.
Parent obviously speaks about common text treated with regular string/text manipulation functions (what 99% of Unix shell pipe work is) vs text formatted and interpreted as an S-expression structure.
The same way a structured english sentence is more expressive than a bunch of vocalizations (even if a sentence is just a bunch of vocalizations too -- it's the structure that makes it expressive).
It's not about the merits of plain text as a general format vs a specific plain text variant. It's about the merits of a specific structure, vs the kind of text files tools like sort/cut/tr etc were made to operate on.
1. LISP 2. Haskell 3. Erlang 4. Something else
No wrong answer, just a question I always like to hear answers to.
Clojure as the most useful functional programming language.
Ditch both if/when you run into their limitations. You’ll then discover various degrees of FP in most languages.
F# is a great language but I've found that it's usable when you can 1) convince C# people it's not that scary, and 2) convince them that you're not An F# Person.
But with just that first Standard ML course, you can already branch out to more popular languages such as Scala [1], which gives you the closest thing I've seen to "Standard ML on the JVM". But Standard ML (or Ocaml, which is very close) might even be your preferred language for prototyping [2], or compiler writing [3].
P.S.: To make Coursera tolerable, I usually pull the entire course with this nice little script [4]. Works great with ranger and mplayer!
[0] https://www.coursera.org/learn/programming-languages
[1] https://www.coursera.org/specializations/scala
[2] https://sanderspies.github.io/slides/dawn-of-reason.pdf
Others have mentioned Elixir, which would also be an interesting option to explore. The main difference being that with Elixir you're leveraging Erlang and BEAM (which gives you a fantastic platform for actors and lightweight threads), versus Java and the JVM in the case of Scala (which gives you access to all of Java).
That's a little meta, don't you think?
I'll probably catch some flack for this (from both sides): You can (and should) use Ocaml as a Lisp where you lose the easy macros but gain strict typing. It's brilliant. The only downside is emacs; it's the only sensible IDE.
Haskell (and friends) are fantastic, but in my experience you have to be reasonably smart and work full-time with it, since every medium sized project gets turned into its own DSL (more so than with other languages). Maybe it's not ideal for consulting.
Amazing ML-derived language with all the benefits of the .NET ecosystem. Highly suggest everyone dip their toe in.
Your problem is not what language to learn, your problem is that you think that learning is "wasting time."
> a functional language in 2019
Lisp is not a functional language, so your question does not even make sense.
If you actually want to help your clients, instead of selling them on bogus, inappropriate solutions (which, with your attitude and lack of knowledge, I guarantee is what you are doing to your clients today), you need to start actually learning the fundamentals of your field. If you think that learning is "wasting time," then you are not learning, you are, indeed, only "wasting time."
I am also a consultant, have worked with all the languages you listed, and I say the above to help you, and other people who consult, and me. Consulting is all about having a very solid knowledge base, continuous learning, and professionalism (which, BTW, includes basic writing skills, like capitalization). Attitudes like yours are why consulting engagements fail, and why consultants get a bad reputation. That only makes it harder for all of us to get work.
:-)
I didn't mind the Trump stuff. I think Scott Adam's insights were valuable even if his comments were extremely frustrating and contradictory.
But his somewhat new Climate Change denial stuff finally made me unfollow him.
https://twitter.com/ScottAdamsSays/status/109968558214017024...
I don't follow Scott Adams, but my understanding from when he started talking about that subject is that he is not denying climate change, but he is acting as a climate skeptic in order to bring about debate and enlighten people on all sides.
For a (possibly bad and certainly trite) metaphor, it's somewhat like saying "Gravity is settled science" to a gravity denialist. You could instead provide some facts and experiments they could try such as holding their hands above their head to fight the force[0] of gravity. They could decide they don't think your facts are correct and perhaps theorize other reasons why they cannot hold their hands above their head indefinitely.
That's science, where even gravity is "just" a theory[1].
[0] Apparently in the Theory of Relatively, gravity is not actually a force, but a consequence of the curvature of space-time and uneven distribution of mass
[1] https://en.wikipedia.org/wiki/Gravity#Modern_alternative_the...
Apparently this is not allowed in polite company.
But I'm saying this as a long time super fan... that isn't what it feels like to follow him anymore. I think he is less passionate about entertaining people or enlightening people, and more interested in bolstering his conservative credibility.
I don't know why his career arched in that direction, and its his right to do whatever he likes. But for me I have had to jump off the train because doesn't seem to be tied to what I liked about his humor.
I'm the conservative version of an Al Franken fan. We came for the comedy, but at some point it isn't about that anymore.
We don't need debate on climate change. There are not "sides". (Unless you count reality and magical thinking as "sides".)
If we aren't debating on climate change, how do we decide where to put resources to reduce emissions and clean up existing pollution?
Questioning the evidence of climate change is de facto adversarial. The evidence is overwhelming and beyond question. No intelligent, sane person arguing in good faith would argue otherwise.
I have not invested enough time in understanding the currently best climate models. I must, therefore, take a skeptical approach and question conclusions drawn from that evidence. Especially so for a projection.
For the level of assurance you seem to have, I assume that you have rerun models yourself, have a deep understanding of complex systems and statistics, and done other climactic work? Even that is skepticism applied and questions asked.
To avoid an echo bubble, don't unfollow people for such reasons. In fact find more people with opposite views to follow.
I say, we should allow huge diversions from our worldview.
In fact, even if its nonsense, if its influencial we should study it and understand it (that is, we don't have to suffer any random nonsense, but if something is both nonsense and big, the real active citizen should study it at least, even if just to try to grasp why their fellow people fell for it and what consequences it might have).
Of course if they start with "they fell for it because they're bad/stupid/crazy/lazy etc" that's just something to make themselves feel better, and doesn't really explain everything.
First of all, this or that people can be bad/stupid/crazy/lazy: those are characteristics of the individual. Millions of people can't be said to be that, there are other forces at play when masses make decisions or adopt things, not personal pathologies.
E.g. it's surely it's not because all the "stupid/crazy" people are in the fly-over states that Trump was popular there. And vice versa, it would be improbable that most of the smart/sane people are in the coasts. (You could reverse this with how MAGA supporters view Hillary supports, and it would be the same argument).
Perhaps its a cultural thing that made this people vote this way and the others that. Or a historical thing. Or an economic shift that's felt in one area and not in another. People should strive to understand those things then, instead of assuming the others are into "nonsense".
Of course people conveniently call their fellow people good/sane/etc when they agree with them (e.g. a vote goes how they liked) and then explain the opposite result at some later point (e.g. people voting something they don't like) as "people being bad/crazy/duped/etc"). They're not bothered to see the contradiction in calling the same masses both things at different times.
This is not about people with another opinion being bad/stupid/crazy/lazy. Maybe they are right and I am wrong. It is about repetition, and myself focusing on stuff I don't know yet.
You don't need (and I didn't ask) to learn "something from everyone who has a different worldview".
But you (or, at least, an active citizen) needs to learn why tons of people share the same different worldview.
E.g. why is there a dissent on "climate change" being real. And the circular non-answers to those questions, like "because they follow stereotypes" or "because they're dumb" are not insightful.
However, reading the 1000th climate change proponent or opponent won't help in this goal, "because they follow stereotypes".
And I never claimed to explain the different world views with "because they follow stereotypes", I explained my own behavior with this statement.
Finally, "because they're dumb" is not insightful. But "because they were lured by simplifying propaganda, and this is because they haven't learned to analyze complex situations" can be a valid explanation.
It's not necessary to completely cover yourself in crap, but you should have an exposure to it once in a while.
This mindset is also how we got the C preprocessor...
The C-preprocessor travesty is text, but hygenic macros are also text. The difference is structure.
You are in a scripting language REPL where you can have a hashtable of words and their counts, and keep it structured
gc words.txt |% { -split $_ } | group | sort count -desc | select -first 5
gc words.txt |% { -split $_ -match '^\w+$' } | group | sort count -desc | select -first 5
Here's my cmdline:
# cat file | preserve only spaces and letters (remove punctuation) | convert to lowercase and put each word on one line | use awk maps to keep a count for each word and print map in two columns, count followed by word | sort in reverse numerically by count column, then by word ascending | print top 10 words by frequency
cat input.txt | sed -e 's/[^a-zA-Z ]//g' | tr -s ' A-Z' '\na-z' | awk -F ' ' '{ if($1 in count) { count[$1]=count[$1]+1 } else { count [$1] = 1 } } END { for(k in count) { printf("%d\t%s\n", count[k],k) } }' | sort -k1,1nr -k2,2 | sed -n -e '1,10 p'
Output:
27 the
21 to
15 a
11 is
10 of
10 that
9 but
8 text
7 was
6 and
Object.entries(document.body.innerText.replace(/[^a-zA-Z\s]/g, '').toLowerCase().split(/\s+/).reduce((o, word) => {o[word] = (o[word] || 0) + 1; return o}, {})).sort((a,b) => b[1] - a[1]).slice(0, 10)
Plain JS is missing some niceties, but being able to crunch some data from a website using the same iterative approach as a shell command line has its place!
{ if($1 in count) { count[$1]=count[$1]+1 } else { count [$1] = 1 } } ,
{count[$1]++}
Why bother 'changing' the input field separator? No need. By default it's already space.
awk '{count[$1]++} END {for(k in count) print count[k],k}}'I wasn't sure what awk would do if there was no entry when it first encountered a new word.
And yeah, I left the -F in there by mistake.
But I don't get overly fussy with pipelines like this. I usually just hack them up as a one off then forget about them.
first sed --> tr -dc 'a-zA-Z \n'
second sed --> headAs with most programming, there are usually many ways to solve a problem. Sometimes the different approaches are equally good, but sometimes the approaches are more or less ideal given the constraints.
There's no real magic, and it's really not difficult. To me, problem solving like this, and ending up with a "script" (just a chain of piped commands with some args) is often the easiest way to solve problems.
In fact, the majority of my Ruby scripts for systems work are combinations of the language and %x'd unix commands. I find it's incredibly powerful.
I think the main thing that stands out to me here is just how long ago it was; Knuth was writing just past the assembler era. None of the modern infrastructure was available, such as languages with builtin dictionaries like Perl and Python.
However, software compositionality is one of those things that's been tried in so many not-quite-successful ways. Pipes? Objects? Interprocess object systems (COM/OLE) or object brokers (CORBA)? RPC? Microservices? Packages? Object pipes (powershell)?
The code I was given was several hundred lines of python. All I did was replace all but the final parsing function with a single "grep | sed | awk | grep | uniq | sort | grep" line and called it a day. I'm sure I could have taken it farther, but for the amount of effort it was a fun little exercise.
I guess I am more curious about what Knuth did! Did he just comment it a lot for "literate" code?
As always I feel the Python (or Ruby if I could write it) is the sweet spot between being easy to read and easy to write.
perl -lne 'map { $words{lc $_}++ } split /\W+/,$_ }{ print "$words{$_} $_" for (sort {$words{$a}<=>$words{$b}} keys %words)' Counter(s.split()).most_common(k)
(A slightly better comparison may be with the source code of the Unix programs (tr, wc, etc), rather than with shell pipelines composing those programs.)
Read the original sources for more:
https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-...
https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-...
It definitely takes a bit of work, but hopefully it's still a lot less work than writing the verbose solution.
The hard part of mastering unix one-liners for me are two things:
- Figuring out which tool can pick apart or reconstruct a specific syntax robustly. I think I spend more time figuring out input parsing and output presentation than anything else.
- Knowing when the right time is to ditch the clever one-liner and upgrade to a real program. There's a benefit to prototyping as a one-liner, but there are cases when replacing the result with a long program is appropriate, especially in a production environment. But, then again, only if the one-liner isn't robust or scalable enough, and usually it probably is.
A verbose solution can be deconstructed and understood by a newbie.
A oneliner? Not so much, if at all.
Good POSIX one-liners don't need to change usually and aren't hard to understand. A verbose program, on the other hand only lasts if it's like K&R C with zero dependencies. All the languages and libraries have changed considerably, scripting languages and versions come and go, C++ is quite different and might not compile later, build systems have backward incompatibilities.
Plus, a one liner is, like, one line. It's really not that hard to understand unless it's intentionally crazy, and even then, even if it takes a half hour to work through, that's still less work than something like Knuth's 15 pages of code.
We could be imagining pretty different scenarios, though, so feel free to share examples. Normally a pipe consisting of a combination of sort, cut, head, and uniq is trivial to grok, and it would be very wasteful to re-implement that kind of thing on your own. Here are a few one-liners I've made use of recently that I love:
# count the function calls in log.txt
grep -Po '^\K.+(?=\()' log.txt | sort | uniq -c | tee callcounts.txt
# sum the timings
# input: 2 columns, time & name. output: 3 columns: time_sum, name count, name
# It's basically a pivot table in one line of code
< time.txt perl -lane '$k{$F[1]}+=$F[0]; $c{$F[1]}+=1; END{print "time\tcount\tcall"; print "$k{$_}\t$c{$_}\t$_" for keys(%k) }' | sort -n | tee time_grouped.txt
It seems for every concise one-liner that requires no internet access there is a complimentary website someone has set up to perform this same task "for free" on a remote computer and return the results in a web page.
I never use those because I still enjoy the command line. People who design software and websites today, and those who review their work in the media, often refer to the command line as some sort of purgatory. God help the user who is faced with a command prompt. For me, this is more like a haven from the world of gratuitously graphical software and web applications that are too often either bloated, overly complex, inflexible, opaque, slow, or some combination thereof.
It is not that the command line is pleasing in an absolute sense. It is only pleasing in a relative sense, when contrasted with the alternatives. If the alternatives are painful enough, then moving to a command prompt is a relief.
I don’t think there is anything fundamentally different between when your tools are Unix cli tools or a programming language. It is the same skill.
tr A-Z a-z | tr -cs a-z '\n' | sort | uniq -c | sort -rn | sed ${1}q
tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q
McIlroy: "A first engineering question to ask is: how often is one likely to have to do this exact task’? Not at all often, I contend. It is plausible, though, that similar, but not identical, problems might arise. A wise engineering solution would produce—or better, exploit—reusable parts." [1]
[1] https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-...
Yours says:
Convert to lower case
Convert non-letters to CR
Convert non-letters to CR
Convert to lower case
sort -f | uniq -ic
for x in `cat foo.txt` ; do echo $x ; done | sort -f | uniq -ic | sort -rn | head -<num>And please be mindful of looping over user supplied data like that. Those things are typically the ones that works fine in test but blow up in production. The environment has a maximum size, and you shouldn't store large amounts of data there.
Split the stream of text into words.
Normalize with respect to case.
Accumulate a count of each distinct word.
Sort by the counts.
Print the number needed.
Perhaps what makes it rare is not having seen an example before - but one example is enough to demonstrate the principle, thanks to its elegant simplicity.
The hard part used to be in remembering which utility has an option that performs the transformation you are looking for, but practice helped with that, especially once one knew enough to guess which utility probably had that feature, at which point one brought up its man page. These days, of course, it is usually easy to search for the solution you want.
$ cat test.file | tr A-Z a-z | tr -cs a-z '\n' > diffSecond
$ diff diffFirst diffSecond
$
No differences with my test file.
I'm something of a command-line wizard (well, given enough time, if I'm honest! I always have to re-remember things like awk syntax), but in interviews now, no one asks core Linux shell-type questions (or even theoretical kernel-type questions). They primarily ask:
- coding (70+%)
- diagramming infra type questions
- experience with keywords ("CI/CD" and "containerization" being the biggest keywords).
Recently I interviewed with a 'startup' that's in the real estate space and I was told by a manager, after touting my Linux infrastructure skills, "Oh, I don't even care about that anymore. I can teach all there is to now in a couple of days. It's all AWS and tooling here."
This is in the DevOps space.
Sounds like the new generation can give 0 craps about the command-line?
Also works the other way around: How many binary trees are you balancing each day?
perl -pe 's/[\s\.,]+/\n/g'
Of course this raises the philosophical question of whether a "one-liner" is in fact a one-liner if it has two lines.
A key skill in good programming is choosing tools at the correct level of abstraction for the problem. Here, the higher-level unix tools are clearly the better choice.
Saying "that's cheating" because the high-level tools are themselves written with lower-level tools with many LOC is missing the point.
He admits they don't compare, but does it anyway. I shouldn't take offense... but for some reason I do!
Unless you're planning on writing raw assembly, you're using someone else's program/work anyway.
https://jugad2.blogspot.com/2012/07/the-bentley-knuth-proble...
Post has rough Python and shell solutions to the problem Bentley proposed.
The solutions can be useful for beginners to Python or shell.
It's really valuable to know awk, but it's a bit misleading to claim that "it's shell".
>It's really valuable to know awk, but it's a bit misleading to claim that "it's shell".
It's more than a bit misleading to accuse someone of being misleading, without checking your facts first.
Maybe you spoke too soon, without reading the full post, as I find people sometimes (often?) do (not just in reply to my comments, but to those of others too), not just on HN, but on many forums.
Read both the header comment and the last line of the script you called "an awk solution":
Header comment:
# bentley_knuth.sh
Of course, the filename extension does not make it a shell solution instead of an awk solution, but I used .sh because I knew what I was doing [1]. See next point.
Last line of that script:
' < $2 | sort -nr +1 | sed $1q
The $2 and the $1 - in the $1q bit - are shell command line parameters, because this code is invoked from the shell as a script, with arguments passed. Also see the pipe symbols (|). All these are part of shell syntax, not awk syntax (although awk has $1, $2, etc. too, they have a different, though related meaning - in fact, I use a $i in my awk script within this shell script too). The whole script is a pipeline, that pipes the awk script's output to sort and sort's to sed.
I don't know if you are a shell/awk newbie or not, but regardless, you missed those points above, like I said, probably due to haste.
[1] Check out this recent article by me in Linux Pro Magazine:
http://www.linux-magazine.com/Issues/2018/217/Exploring-proc
for a bit of shell quoting magic along with some use of awk.
And this post:
UNIX one-liner to kill a hanging Firefox process:
https://jugad2.blogspot.com/2008/09/unix-one-liner-to-kill-h...
and also the interesting comments on that post, from which I learned some things.
After having worked for years on Unix platforms (as both a dev and system engineer), from even before Linux was created, I think I know the difference between shell and awk (and a bit more, although still do not claim to know everything, or even close).
On a more positive note, I'll use this as an opportunity to put in a plug for my Python and Linux training offerings :) Course outlines and a couple of testimonials here:
Quoting from your post:
And here is my initial solution in UNIX shell:
# bentley_knuth.sh
# Usage:
# ./bentley_knuth.sh n file
# where "n" is the number of most frequent words
# you want to find in "file".
awk '
{
for (i = 1; i <= NF; i++)
word_freq[$i]++
}
END {
for (i in word_freq)
print i, word_freq[i]
}
' < $2 | sort -nr +1 | sed $1q
You're doing all the word counting in awk.
Yes, you're invoking awk from a shell script, but that's really not the same thing as "using shell." McIlroy’s solution is genuinely shell:
tr -cs A-Za-z '
' |
tr A-Z a-z |
sort |
uniq -c |
sort -rn |
sed ${1}q
Perhaps you don't agree, perhaps you think "awk" is a command line utility. If so, then we'll agree to disagree.
Meanwhile, you might want to scrutinize your own reply (the one to which I am replying here) and think a bit more deeply about what might be wrong with it. And until my full reply to come later, here are a couple of hints:
Hint 1:
>"awk" is generally accepted as a full programming language, whereas "tr", "sort", "uniq", and "sed" are command line utilities.
- a tool can very much be both a full programming language as well as a command line utility at the same time. awk falls into that category [1], as do many other Unix commands. Who made up a rule that it cannot be both at the same time? You?
Hint 2:
Check out your line:
>So you invoke awk, and then run the output of awk through sort and sed.
and compare and contrast its meaning with the meaning of your few lines immediately below it, including the one that says "McIlroy’s solution is genuinely shell:". Try to see the similarity/difference/contradiction.
[1] Finally, read the book The Unix Programming Environment, a classic, by Kernighan and Pike (Unix pioneers). I cut my Unix teeth on it, years ago, although, of course, years do not mean I am right and you are wrong. Facts do. There are chapters in the book on awk and sed. And IIRC they come under the topic of filters (maybe even the chapter name is that) a.k.a. command-line utilities, although not every such utility needs to be, or is, a filter. I think you have some confusion about terms and their meanings, and/or are assigning your own meaning, even though you use words like "generally accepted".
Also skim this article (published by me, years ago, on IBM developerWorks) to get your fundamentals more clear:
Developing a Linux command-line utility:
https://jugad2.blogspot.com/2014/09/my-ibm-developerworks-ar...
Enough for today - will do follow-up comment as I said, if needed, in a day or two.
Thank you, by the way, for your references to various published material. FWIW, I've worked with BCPL, C, AWK, C++, Unix, Linux, GNU, and much, much more, for the last four decades or so, so I'm not inexperienced, and I have read most of the classics. That also doesn't mean I'm right, but it does mean that I have a basis for my opinions.
So thank you for your offer to school me, but I'll decline, and, as I say, accept that we disagree.
Thanks for casting aspersions without even so much as a mention of what the "so many points" are that I am supposedly wrong on.
When I said upthread that you have misunderstandings, I at least mentioned some and hinted at or gave a clue to what the others were.
Also interesting that when kazinator said to you that awk is part of shell, you meekly accepted that he was right, thereby contradicting your earlier claim that my shell solution was not a shell but an awk solution. And in that same comment ( https://news.ycombinator.com/item?id=19279030 ) accepting it, you still seem to be neither here nor there, by your own words, where you say things like you "see why it could, perhaps should, be though (sic) of as being "shell", but "always have trouble thinking of it as such".
https://news.ycombinator.com/item?id=19276012
Interesting and maybe significant that you say: "Then we'll agree to disagree. I think you are wrong on so many points here, it's clear we're not going to agree, and probably won't find common ground."
First, interesting that you say "I think you are wrong on so many points here ..." but do not deign to offer any points to back up your statement. Kind of a cop-out, looks like. Anyone can say someone else is wrong; such statements do not carry any weight unless backed up with something more substantial.
And about your "four decades", like I said in a previous comment, years or age do not matter, facts do. I care not a whit if the person I am arguing with has 4 years or 4 decades or 4 centuries of experience. They (or I) can still be wrong (or right) about any specific topic we happen to be arguing about. I've been known to acknowledge that I was wrong, in arguments with people less experienced than me, many times, and vice versa has happened too.
Nor is finding "common ground" the goal (this is not some sort of compromise between political parties, it's a technical argument). Getting things right is the goal. For which, sometimes one party or the other may have to admit they are wrong - including me. Just that I do not think I am wrong in this case.
Will still write my fuller reply as I said earlier, to keep my word, and to make the picture more clear for other readers, since you have made these statements, even if you have hastily left the conversation.
> Nor is finding "common ground" the goal (this is not some sort of compromise between political parties, it's a technical argument).
We disagree. When there is a disagreement, finding what you agree with the the first step in finding where the lines of reasoning diverge. Finding common ground is the first step in resolving differences.
> Getting things right is the goal.
Sometimes in software there are judgement calls. Maybe this is one of them, maybe our definitions differ. Sometimes definitions differ because of context or experience. In each case, the terms used are not right or wrong, they are definitions that are useful in the context.
> For which, sometimes one party or the other may have to admit they are wrong - including me.
This is not an "I'm right, you're wrong" situation. By my experience, in my context, what you wrote would be called a "shell solution" in the same sense as the original command-line solution would be called a "shell solution."
You think that invoking AWK from the command line means that it's still a command-line script. Your definition of the terms means that you accept that invoking AWK still lets you call it a "shell solution."
I think that is fundamentally and structurally different from using command line utilities such as tr, sed, sort, and uniq.
So my position is clear - your solution that you call "shell" is not, in my opinion, just "shell". To me, your solution is an AWK solution, and you feed the output from your AWK program through shell utilities.
You are using the terms in a manner that is different from how I'm using them, that much is now clear.
Do you agree that you have written a shell script that invokes a program written in AWK?
Would it be different if you wrote a shell script that invoked a C program by calling a C interpreter? Would you still call it a "shell solution to the problem?"
Does it matter? Really? I've made clear why I've said that I don't class your solution as being shell, why do you care?
So it's shell. I might, however, given my background, and remembering as I do its first introduction, always have trouble thinking of it as such.
I guess I was triggered by the fact that the proposed shell solution is:
* not on a command line (although it could be),
* is significantly longer than the original command line solution, and
* gives a different result.
But you're right, it's shell. I might, however, given my background, and remembering as I do its first introduction, always have trouble thinking of it as such.
If you consider awk "not-shell" because it's an entire language, then it's really inconsistent to consider sed "shell". sed is a stream programming language. For example, this is a Sudoku solver written in sed: http://sed.sourceforge.net/local/games/sedoku.sed.html
I remember when awk was first implemented. sed was already standard, and awk was this new thing. I love it, and for some things it's my "go to" language. That colors how I think of it - I think of it as a language.
But this has been done to death, everyone is jumping on me, so there seems little to add.
No one is "jumping on you", as you put it, neither the other people who replied to your comments, nor me. It's absolutely normal in a tech forum (more so in such ones, because they are mostly fact-based) and in fact even in any online or real-life forum, for that matter, for people to point it out if they think some statement a person has made, is wrong.
In fact, you did exactly that (point out that you thought I was wrong - see your comments that I've quoted just below), in your top-level reply to my original comment about my having created solutions in Python and shell, which is what started this whole sub-thread. Going by your logic, I should have complained about you jumping on me :)
Here's where you pointed out that (you thought) I was wrong:
>Your "shell" solution isn't really a shell solution, it's an awk solution.
>It's really valuable to know awk, but it's a bit misleading to claim that "it's shell".
Not only that, you claimed that it was misleading, without having any way to know whether I had any intention to mislead or not. Come off it. You cannot read my mind. It's mine, not yours. Come to think of it, that was a poor judgement call on your part, too, because even if I had some intention of misleading people, which I did/do not, what could I possibly gain by misleading them about whether some code is a shell solution or an awk solution? The whole "misleading" idea is a figment of your imagination, or of your unclear thinking, I'm sorry to have to say.
Anyway, this thread has gone on for too long, with barely any benefit to anyone. I'll just briefly touch on that fundamental flaw in one of your points about my work, that I mentioned in another comment, and then be done with this whole thing. Doing that in a separate comment.
Google for "sed is Turing complete" and see the results, including the post at catonmat.net - Peteris Krumins' blog :) There's even an HN thread about it.
Whoever authored that, mad props.
I this read you other replies and I can't see them adding much value to the discussion.
You are just being rigid about semantics.
What action are we supposed to take having read your comments?
To be careful of not calling a pipeline with awk in it 'shell?'
To be ashamed of using awk because it's a programming language, and that is 'cheating'?
Thank you for the feedback.
I said I would point out a "fundamental flaw" in your points. The flaw is this:
You thought (and said) that my shell solution was an awk solution. That is wrong. It is a shell solution (and not an awk solution) for multiple reasons, which any slightly-more-than-beginner-person to awk and shell, should have easily known, if they had their fundamentals clear, which implies that you do not. It is a shell solution because:
1) the entire script is a pipeline (which is obvious to see from the pipe signs used, if you knew your stuff and had paid attention to the code, before writing your first reply). awk does not have the pipeline operator (as meaning send the output of the previous command to the input of the next command). That itself should have told you that it is a shell script, not an awk script. There is an awk command embedded in the shell script, but that is very different from saying that it is an awk script.
2) You said elsewhere in this thread, in reply to kazinator:
>That I did not know - thank you. We here still think of awk as fundamentally different from other pipeline facilities such as tr, sed, sort, uniq, and so on, but I can see why it could, perhaps should, be though of as being "shell".
That statement of yours above is wrong on two counts:
a) awk is not fundamentally different from tr sed, sort, uniq, etc. It is a Unix command-line command like any other. The fact that it happens to be what you and some others may call a full programming language (not a well-defined term, anyway) does not make it any less of a command-line command. A tool can be both of those at the same time, and awk is. So is Perl. So is Python. So are many other languages. In fact as someone else said and I hinted at, sed may be a Turing-complete language. So does that suddenly make my script a sed script, just because I used sed in it? But I used awk in it too. So should it be called an awk-sed script? But I used sort too. So now should I call it an awk-sort-sed script? See what I am getting at? No, it should just be called a shell script, because that is what it is. The shell is a high level language that orchestrates other programs via its syntax and operators. (See below about the shell's operand being whole programs.) You claimed that the main work of my script (the word counting) was done in awk, and the results piped to other commands, therefore it was an awk solution. But it is the shell that is doing the piping, not awk! awk cannot do such orchestration, at least not easily, not without resort to the "system()" library function it has, but that is again implemented using the shell (and other stuff, like fork and exec system calls - I'm simplifying here).
All shell scripts can consist of any command or combination (not just pipelines [1]) of commands, irrespective of the type of the command, whether it is a programming language or not, what language it is written in, etc. In fact there is not even a requirement that the commands used in shell scripts should all be filters; that requirement is only for shell pipelines. [2]
[1] A shell script can: consist of just a sequence of (one or more) command(s), terminated either by semicolons or newlines, or both; consist of one or more pipelines only; consist of any combo of the preceding. And also other variations, including at least an ampersand (&) terminating the command (or pipeline), which makes the preceding command or pipeline run asynchronously from the rest of the overall command/pipeline/script, if any, i.e. in the "background", as we say in Unix.
[2] Here is a shell script that demonstrates many of the above points:
# a_script.sh
foo1 # run foo1
foo2; foo3 # run foo2, then foo3
foo4 & # run foo4 in the background
foo5 > f1 #run foo5, redirect its stdout to f1
foo6 < f1 | foo7 arg1 arg2 | foo8 arg3 arg4 arg5
All this is part of the flexibility, beauty and power of the shell, whether used in scripts or on the command line. Which brings me to my next key point: there is essentially (almost) no difference between typing commands interactively at the shell prompt, and invoking the same commands from within shell scripts that are run by the shell. The exact same syntax with the exact same semantics can be used (for all practical purposes, maybe with a few exceptions, in both modes, interactive or script).
In fact, you can even type for and while loops [3] (including with redirection of their input and output) at the shell prompt. (You can even type if statements at the shell! Same for case statements.)I do it all the time, for throwaway "scripts" such as ones to monitor the execution of some processes, and so on. And many standard Unix books - like classic book, The Unix Programming Environment (UPE), that I mentioned in this thread - show that in examples.
Another thing that UPE says and shows is something to the effect that "the shell is a very high level language - its operands are whole programs (emphasis mine)". That is why we can do things like the example in [3] below, but first, another example:
while : # : is a built-in that evaluates to True,
saving having to run the true command from disk each
time
do
ps -aef | grep foo
sleep 10
done
[3]
while ps -aef | grep foo
do
# something or just sleep a bit
done
b) You called tr, sed, sort, uniq and so on, "pipeline facilities". They are that, but are not just that. Before and apart from the fact of being "pipeline facilities" (which is anyway, a non-standard term you used, a better and more standard term would be just "Unix commands" or "filters" - filters is a standard term, for programs that read either filename arguments or their standard input, process the input in some way, and write the results to standard output, thereby enabling the whole Unix pipeline paradigm), they are also simply normal commands, or programs. Any of those commands can be used either standalone, or in a pipeline. In fact there are other ways of using them too, for example, you can invoke any of those commands (as well as any other executable) as a child process from some other program you write, in C, Python or other programming language. You are creating distinctions where none exist, for who knows what reason.
(Or maybe someone wrote a 1 liner for that too?)
Personally I prefer to write small bash scripts over one liners so future colleagues can understand the program's logic.
Thanks for this!
[msimmons@msimmons-lin ~]$ time cat moby10b.txt | tr " " "\n" | grep -v '[[:space:]]' | sort | uniq -c | sort -g | tail -n 10
1592 I
2122 his
2446 that
3462 in
3906 a
4051 to
5225 and
5514
5851 of
11997 the
real 0m0.768s user 0m0.805s sys 0m0.021s
It lives up to its name and is very much a place to start and nothing more.
Your real problem is going to find enough problems to solve to build up muscle memory.
When I was starting out, I deliberately took the time to truly master as many of the UNIX tools as I could.
When I would run into a problem like this, I would use it as an exercise to get better at them.
In most cases, especially the first time I used each tool, it took much longer, but it paid off in the long run.