The Shell Hater's Handbook (2010) [video]
confreaks.tv
confreaks.tv
It took a lot of pushing from a mentor for me to finally click with the overarching principles of shell and see how to master it and become productive with it. He really forced me into it as well, I wasn't a willing pupil for quite some time, but he kept persisting and finally I started listening to him. I'm so glad I did. I've never read any book or tutorial or guide that teaches what he taught me either, despite searching several times. Also, I don't think I could have learned to master shell if I hadn't learned functional programming first. The day I realised how amenable shell is to higher-order thinking was a real epiphany.
These days shell is a really important tool in my toolkit, and I can see just how much some of my colleagues are held back by not knowing it.
Anyway, I really enjoyed the video, it was a nice exposition.
- I used to spend a lot of time stuffing things into variables and then trying to operate on those, which is how you do things in imperative languages. Shell is better thought of in dataflow terms, i.e. pipelines. So I started using pipelines more and more.
- The corollary is that you start thinking more about data (passing between pipeline stages) and less about control flow.
- Compare an if statement in C, for example, to an if statement in a functional language. In C an if statement has no value per se, but in functional languages if "statements" are actually expressions that have a value:
f :: A -> String f x = if isFoo x then "foo" else "other"
In shell an if statement is just another command, so you can do things with its "value", i.e. its stdout:
if do_test
then
cat file1.txt
else
cat file2.txt
fi | grep pattern
Because I was so used to thinking of if statements as just control flow, I never used to think of piping them to another command like that.Also if you think carefully, that last pipe to "grep pattern" has the flavour of partial evaluation in functional programming.
- I tend to think more in terms of building up an execution environment these days, in the same way that lexical scope works in regular programming languages. In shell, of course, that means command invocations, sub-shells, etc., because the shell doesn't really have good lexical scoping. The simplest example I can think of is executing some commands in another working directory with a sub-shell:
(
cd $dir
run_test > "${f}.log"
)
You could compare it to RAII in C++ or with- style macros like WITH-OPEN-FILE in Common Lisp.Those are just some examples.
EDIT: Also, have a read of the links in chubot's post: https://news.ycombinator.com/item?id=14012453 . That's the sort of mentality that will change your approach to shell programming.
Perhaps, but the shell is still a badly accrued (more than designed) set of options.
Something like ZSH and the modern Fish shell showed how much traditional shells can improve, but we also need some better primitives, and more controlled experience.
Oh, and they should NOT be based on emulating 40+ year old teletypes anymore, except as part of a "legacy" mode.
I wouldn't mind a better shell, but I'm not going to throw out my current shell for the lack of a better shell.
https://www.gnu.org/software/bash/manual/html_node/Shell-Par...
That's the sort of thing that turned me off. I still tend to avoid those nooks and crannies, although they are sometimes useful. I'm not an expert.
I'm not sure about Unix shells being inspired by C semantics. I actually started out on csh or tcsh, I forget which, two decades ago. I'm not sure that making a shell language "C-like" is a useful goal, and those two shells both fell out of favour because they have many flaws. I switched to bash quite quickly.
I got past all of that, and I also see people struggling to accomplish things that can be done in 30 seconds with shell. My main language is Python, and for certain tasks it just can't compete with shell. Believe it or not I think I once wrote a Python script to do what "find | sed -i" does.
Now I'm back in the "who would design such a silly language" mindset, and I am designing and implementing a new shell. You might like some of my blog posts:
"Pipelines Support Vectorized, Point-Free, and Imperative Style"
http://www.oilshell.org/blog/2017/01/15.html
"Shell Has a Forth-like Quality" http://www.oilshell.org/blog/2017/01/13.html (on HN previously)
I guess maybe because I started with computers before there were GUIs that is what felt normal.
For me, probably not. I did use DOS before Windows as a kid, but that was long before really learning to use a shell. Familiarity with DOS simply made me familiar enough with the idea to be unafraid of it.
When I learned to manage files, handle software, etc. I learned to do it with a GUI in Windows. When I started to use Linux, I was vaguely aware of the usefulness of a shell, and took the opportunity to really learn. To this day, I rarely even have a file manager installed, in favor of a shell environment.
I also got my start on pre-GUI computers (a VIC-20, specifically).
Multiple contradictory answers, disagreement, sub-sub-sub explanations, historical context, caveats, workarounds, hypothetical scenarios, portability, personal taste, different versions and implementations all show up. One answerer felt the need to include a footnote.
It's like the opposite of Python's "there's one right way to do it": There are 50 right ways to do it, all of which are wrong in some case or another, potentially depending on which way the wind is blowing.
All in regards to checking for an empty string.
[1] http://unix.stackexchange.com/questions/136628/bash-script-x...
Probably better described as an ideal, not always lived up to, but serves as a guiding principle...
I for one like my shells dirty with hidden corners and black magic. Name anywhere where you do not have bash available; embedded, grub, rescue boot shells. I don't do embedded. Never will.
But on a serious note, does anyone know a good resource where I can pick up a better la guage and never have to bash script again? I love writing it. It I am sooo restricted by it, yet keep solving problems with it and also making good money like last night. Maybe Python is the way?
Python is definitely a capable language for scripting, but I would also point out that Homebrew is built on Ruby and uses that to great effect. You might also check out this article on scripting with Ruby:
http://radek.io/2015/07/13/ruby-scripting/
And if anyone else has any tips on this subject please do pass them along.
Would you care to give a link to your guide? I'd be interested to read it.
* It's a work in progress and a bit lacking in visual presentation at the moment
* The target audience is the novice developer
* I am writing from my own experience and I am not an expert
* "comprehensive guide" should probably have been "comprehensive primer". This is what I think beginning devs should know about the shell.
https://tenebrousedge.github.io/shell_guide/shell_guide.html
Commentary, contributions, and criticism are invited.
As far as 'shipping something today' is concerned, that's what keeps me from learning vim. At the moment I'm both working and going to school full-time, so I feel like I can't quite justify the time investment, but I know I'm holding myself back from being a far more effective programmer.
I also have an acquaintance doing computational biology for whom Bash is an essential part of his toolkit for DNA analysis. Bash sometimes is the right tool for the job. It excels at text processing. It's just archaic and ugly, and we have nicer options available for programming languages.
(and now that my SO is up I think I'll watch this video)
I only skimmed so far, but this quote stuck out:
"Programming is fundamentally a way to save human labor, and that includes our own labor."
I'm continually amazed at how many software engineers seem to miss this point.
I will also add the further caveat for anyone else who arrives here that this was written in stolen hours in the last two weeks, and specifically because the page of 'Assumed Knowledge' referred to in the introduction seems to have been the only instruction in the shell given at Epicodus' Intro to Programming. I'll never love Bash as a programming language, but given that it's a tool developers use every day, I do think there's a certain minimum amount of knowledge required to use it well.
If you want to learn vim (I would definitely recommend you do), learn a little at a time. Vim is better suited to this approach than you might think.
People tend to focus on vim's learning curve, but as soon as you grok 3 things, you can do everything you used to do with normal editors. Then all you need to do is learn one small function at a time.
1. :command
:w save the file :q quit vim
2. Normal mode
You are here. Pressing a key runs a function. Escape cancels. u is undo, Ctrl-r is redo.
Every key has some specific use. You can learn them one at a time.
My favorite is . (the . key). It repeats the most recent action at the cursor.
3. Insert mode
Pressing i puts you in insert mode. Here you can enter text like normal until you press Escape
Remember that everything that happens between pressing i to pressing Escape is one action that can be undone or redone.
Later you will learn that a s r and o all do similar things to i.
For this sort of thing, I will use a combination of Bash, Awk, Perl and Python, with Bash as the foundation, but often using at least one of the others. For example, if a step in a pipeline can easily be solved with regular expressions and/or simple arrays and hash tables, I will pick Perl, but if it needs more complicated data or functions, I will pick Python.
* Python does strange things with IO buffering. Just when you think you've nailed all the edge-cases of this, some more come along. Its decisions are good for a general-purpose programming language, and awful for a shell.
* Python is awkward for some close-to-the metal operations such as interaction with unix sessions and process groups.
* Python forking is wacky. There's magic in there, and there's some stuff that gets inherited under the cover. If you fork and don't immediately execv, you are probably in for Interesting Times.
When I looked at the code for Ansible, I felt a bit like I'd encountered a rival alien civilisation who had tested and reacted to the same laws of the universe, and reached identical conclusions. Ansible is written in Python. But it generates bash scripts, and then bash does dispatch.
Given the power of general-purpose languages we have on unix, it's remarkable how stunted shell evolution is. Which is to say, there is an exciting opportunity here.
A shell built as a Forth dialect could be a game-changer. It would have a certain number of powerful builtins. It may even evolve higher-level programming concepts. I am interested in further discussion towards this, and maybe a new github project, particularly a collaboration with someone with high fluency in Forth. I'm in London: beers.
The art of a good shell would merge (1) fluid interaction; (2) effective wrapping of the unix API; (3) high-enough-level programming.
In an alternative universe it would be nice if all these tools would consume and emit s-exprs. Or maybe I'm just wishing for the return of the Lisp machines, where AFAIK "the shell" was the Lisp REPL (never used one myself so I could be all wrong, of course).
Kids these days would of course use json instead, at least until the serious (tm) enterprise people would swoop in and replace it all with xml and ShellReplFactoryInstanceFactoryFactoryBean.
You forgot to include Adapter, Builder, Strategy and a few other words in that.
JSON? How last week. The cool(est of the cool) kids are already moving on to TOML and its ilk.
The Bentley-Knuth problem and solutions:
https://jugad2.blogspot.in/2012/07/the-bentley-knuth-problem...
It is about an interesting programming problem to do with text processing - involving finding word frequencies (apparently initially posed by Jon Bentley to Donald Knuth).
The above post (on my blog) has two solutions by me, in Python and shell, and links to the original discussion on another blog, where I first read about the problem - which was an interesting read, with many comments:
More shell, less egg:
http://www.leancrew.com/all-this/2011/12/more-shell-less-egg...
I completely disagree. As the sysadmin who has been the guy actually managing devs/programmers messes in prod, shell is the one thing that just works, is easily readable, is easily documented, is fairly easy to write, and generally keeps me from pulling my hair out or become violent with the devs.
Take it from the guy who actually has to wrangle all your strange python/perl/go/node... bullshit, shell is something you shouldn't be bashing.
But apparently, bash is something you should be s(h)elling.