Become Shell Literate
drewdevault.com
drewdevault.com
People like to pretend that learning shell commands is somehow better because it's more portable, but unless your work entails working across multiple machines and environments daily, it doesn't matter. Most of us do most of our work on one machine, in one environment, so at the end of the day, it really comes down to preference. Personally, I do prefer working in the shell, but if you prefer working in a modern IDE, don't let anyone fool you that you're somehow wasting time or being less proficient than you could be in a shell--if you took the time to learn the IDE keybindings and the IDE is on par with anything Jetbrains puts out, you're not losing in efficiency at all and possibly making gains when it comes to certain tasks.
No, it's better because if you aren't doing exactly the workflow an IDE or GUI tool designer has envisioned, it is almost invariably much easier to do it in shell (and then make it a script and then bind it to a command in your IDE or GUI tool, if they support that) than to beat the non-shell tool into, first, doing what you want, and then making it easily repeatable.
It's also more portable, which, contrary to your description, is very useful for most people, because even if there are some dev-only tasks that “I can only do it in the IDE” is fine for, for many things you may also want to do in a CI pipeline, on a deployment box, or other places where your IDE isn't running. Shell scripts generally work there.
Modern IDEs have shell built-in so when I finish coding, I can just open up a terminal inside IDE to run shell scripts. Effectively I'm getting the best of both worlds.
I never knew that I have to follow some sort of rails laid by IDE designers, which I think it's a very common misconception by people who dislike IDEs.
Even better, because IDE integrated shells are many cases graphical REPLs with additional interaction capabilities and graphical abilities.
As for the travel into memory lane, here out of 1977,
https://en.wikipedia.org/wiki/Xerox_Development_Environment
You can have a nice view how it worked, by following the Wikipedia links, or checking the presentation on Mesa/Cedar, which was the evolution of XDE as Mesa evolved into Cedar, and one of the latest version of XDE as well.
"Emulating a Xerox Star (8010) Information System Running the Xerox Development Environment (XDE) 5.0"
https://www.youtube.com/watch?v=HP4hRUEIuxo
"Eric Bier Demonstrates Cedar"
https://www.youtube.com/watch?v=z_dt7NG38V4
And naturally Smalltalk-80 and Interlisp-D enviroments that precedded it,
"Emulating Smalltalk-80 DV6 on a Xerox 1186 (6085 Daybreak Development Kit)"
https://www.youtube.com/watch?v=dpjRZnUw8MU
"The Interlisp Programming Environment", 1981
http://larry.masinter.net/interlisp-ieee.pdf
The lack of teaching of the computing world outside Bell Labs leads to a UNIX cult, unaware of the progress that was actually already available in the 70 and 80's, but unfortunely failed to pick up due to several reasons, so in the end there is this idolatration of the UNIX shell.
Sure. In GP I talk about the value of knowing the shell when you use an IDE, not the value of knowing shell instead of using an IDE. To get “the best of both worlds”, you have to know how to use the shell.
> I never knew that I have to follow some sort of rails laid by IDE designers
You don't, if you know how to use the shell. Whether the shell is integrated or external to the IDE is a side issue.
My setup is specifically crafted between my text editor (that has a terminal that I use all the time) and a few GUI tools that I’ve got customized and scripted. I like a mix of shell and GUI. That’s my preference.
You’re right that it isn’t as portable as my dot files repo (and I have one of those too), but I do actually have it set so I can quickly restore the whole config on a Mac and restore the text editor portion from Windows, or really any machine with a web browser.
If you are constantly using different machines on the regular, yeah, being good at shell commands makes sense. But it still comes down to personal preference and people who like other methods aren’t inferior.
I totally agree with that sentiment.
"[...] unless your work entails working across multiple machines and environments daily"
That situation might be a lot more likely than you'd think. I started out doing WordPress, and that's still what mostly what makes me money.
But now that I am using 30-40 different machines that host 3-400 sites, each with their own installs, and knowing how to do stuff efficiently with the shell is super useful.
To me, that situation feels kinda unlikely, but at the same time dang it's nice to just ssh into a machine and know what I'm doing.
I have coworkers and clients who just can't do certain kinds of tasks or troubleshoot stuff with certain kinds of tools because they just don't have the CLI knowledge.
So while I generally agree with your post, I will offer an alternate idea to balance it: if folks are thinking that learning how to use a shell is a waste of time because they can do everything in a GUI, if they took the time to learn the shell at a a level of most of the things they use every day in an IDE, they are "not losing in efficiency at all and possibly making gains when it comes to certain tasks."
It is more portable. No pretending is necessary. It's true. I run multiple computers with different resource constraints and operating systems. I neither have the patience nor the time (not to mention the system requirements) to install an IDE on all of them. However each one has an OS that comes with a POSIX-like shell, e.g., NetBSD's sh, FreeBSD's sh, OpenBSD's sh and Linux's sh, which is derived from NetBSD's sh.
The author of this blog post begins his demonstration of shell wizardry with the "history" command. This command does not exist in POSIX sh. On NetBSD I use "fc -l 0". Go figure, it is more portable (nevermind fewer keystrokes). The scripts I write in NetBSD sh run on FreeBSD, OpenBSD, Linux, and a number of other OS without any modification. I do not have learn multiple shells to do work. On BSD, I also use the POSIX-like scripting shell (sh) as the interactive shell.
The best part is I do not have to install anything, it has already been included, no worries about system requirements. All these OS require a POSIX-like sh. None of them require an IDE.
But now you can use Linux subsystem for Windows. I don’t have much experience with this so I’m not sure how well it works in practice but most people seem to rave about it.
I’ve anecdotally noticed more people using or willing to use PowerShell in some scenarios, now that it’s at parity on Mac and Linux (there are a few things in old-school classic Windows PowerShell that don’t work on Mac/Linux but the future direction and support is all x-platform) and it’s certainly easier to get a Linux shell running in Windows than ever before.
Disclosure: I work at Microsoft but not on any of these tools.
Plus you get a huge standard library out of the box with its .NET integration.
It's really impressive what Microsoft has done with it.
Personally, I don't like the shell that much. As others pointed out, it is a consistency-lacking set of programs that often do only trivial tasks (by design).
A proper cross-platform scripting language such as Perl, Python or Newlisp. The latter is very much like Busybox : it packs a lot in a small single executable (plus it is a Lisp-like language so the syntax is simple and the documentation is good).
Architects also can use Mac's only part of the time. A lot of software just isn't available for anything other than windows (Rhino etc).
1) shell commands
2) sql
3) vi
4) some programming languages c, pascal(?), basic(?)
Everything else has changed multiple times. Maybe that means its worth investing on learning them.
Perhaps if Rust had five mutually incompatible borrow-checkers which get called based on who wrote the code for a particular language construction-- that might get across the frustration I feel when using the shell. (Well, honestly I just StackOverflow for the shell incantation I need and that seems to work well enough.)
You might enjoy https://cheat.sh which is usable via curl: `curl cheat.sh/grep`
For me, reflecting on that history helps dull the annoyance of having to type `grep --extended-regexp` but `sed --regexp-extended`.
> `grep --extended-regexp` but `sed --regexp-extended`
Why not use `grep -E` and `sed -E`?
Sometimes I actually read the POSIX man pages instead of the standard ones — they are sometimes more useful because they have fewer features. Sometimes they are less useful because of their verbose language, though.
That’s probably why DuckDuckGo’s `!man man`/manpage.me default to FreeBSD manpages. But if you really want something non-verbose you may prefer tldr.sh, cht.sh, or bro pages.
I use ag many times a day to search 10,400 files comprising about 800k lines of code, and it is in every meaningful way completely instantaneous. It also understands repository structures, and so won't waste your time there.
It seriously changed my (programming) life.
[0] https://tldr.sh/
That is no way to learn. In foreign language 101, they start you off with a small group of examples. "Como estas?" "Muy bien. Y tu?" Afterward, they explain the rules of the language (this is a noun, this is a verb, this is how you conjugate for first-person singular, etc.). In fact, this is how we learn our first language as babies. Anyone remember Mom and Dad pulling out a flip chart? Or did they just talk to you a lot?
The same goes with apprenticeships, I would think. The blacksmith starts the apprentice with simple tasks around the shop. I suppose he would intersperse it with the occasional pontification about principles and theory, but he would not sit down the pupil for weeks explaining everything before just letting him get his hands dirty.
The Linux man pages are upside down. Examples don't come till the very end, if at all.
Thankfully, like you said, there is now Stack Overflow.
You only learn the tool once. Every subsequent time you visit the man page, the information you're probably looking for is frontloaded. Just scroll to the bottom if you want examples?
The figure on most keyboards is G. Yet when you press it, it puts on the screen, g. Chromebooks are better in this way. Their keyboards are labeled in lowercase.
I actually went back and forth between saying Shift-G, or just G, for this very reason. So I erred on the side of clarity.
In some contexts :help notates things as characters (ex zh, zH, and z<CR>). In other cases I'm seeing things written as <S-F11> and <C-G>. There's also CTRL-H (instead of <C-H>) but I'm not seeing shift written out like that for whatever reason. Sometimes they get mixed (I'm not sure what the rules are) such as for hh<Space> and hh<C-]>. Amusingly enough, :help appears to treat Meta-{char} as case sensitive but CTRL-{char} as case insensitive (I assume there's a reason). I also spotted a <kPlus> (for the keypad).
What an amusingly pointless distraction!
This is false for most tools we don't use daily. It's false even for tools we do use daily, in some cases (I have to google git commands every month or so for commands I use rarely).
Most people are and remain perpetual intermediates: https://blog.codinghorror.com/defending-perpetual-intermedia...
The experts are the exception.
> Of all the professional hubris I've observed in software developers, perhaps the greatest sin of all is that we consider ourselves typical users.
> We are experts. Who could possibly design software better than us superusers? What most developers don't realize is how freakishly outside the norm we are. We're not even remotely average-- we are the edge conditions.
Honestly, if you’re a hobbyist or moonlight as a FOSS contributor, fine, nbd. But if I found out an alleged professional working for me writing software was content to never figure something out like a git branch issue in the short term and grow over the long term in their total skillsets, I’d want them gone.
I wouldn’t want a lazy, mediocre carpenter to build my house either. I’m perfectly content to let them build a ramshackle residence for themself where nobody has to suffer because of it.
[0]: https://blog.codinghorror.com/the-rise-and-fall-of-homo-logi...
This is kind of funny since I'm European and we view most US houses as low quality McMansions :-))
Even then, I lookup the quoting difference between $* and $@ every single time, and how to use `read`. Or more likely decide it's time to drop bash at that point...
Just stop using GUI utilities. It really is that simple. If you just don't use them you'll end up in a shell out of necessity because you still need to get things done.
Of course, the majority of my time is spent in my web browser reading documentation followed closely by vim for writing things. Actual time spent interacting with CLIs is a small minority at the end of the day.
man pages are meant to be a full reference, not a quick tutorial.
To get the quick tutorial others have already mentioned cheat.sh [1] and tldr pages [2] is another good resource.
[1]: https://cheat.sh/
[2]: https://tldr.sh/
And it's fun to read. It's how man pages should be written, at least have a tl;dr before boring you to death.
Yes, and for both quick and longer tutorials, there are also these things called books and courses - both of which are available in hard copy / offline as well as soft copy / online versions from many years now :) Many of us grew up using them and investing in them for our careers ...
It's interesting because we can imagine a history where the docs were written much more professionally with such examples. And we can imagine that work having lowered the barrier to entry such that a critical mass of users becoming compositionally literate in shell scripting. (And perhaps shellcheck being written much earlier in this alternate history.)
But Stack Overflow not only obsoletes such an effort, it IMO obsoletes becoming literate in shell scripting, at least in the way the author describes. SO's existence is equivalent to being able to write a natural language query on the command line which "automorphs" into the relevant Stack Overflow example. At that point you just need to understand basic piping, redirection, and enough of the syntax to spot-check the magic answer in order to make small changes for a use case. That's a different kind of skill.
[1] https://tldr.sh/
You mean like the five mutually incompatible async solutions in rust?
Almost nothing in computerdom was designed from a developer-centric perspective and it blows my mind.
Maybe I'm 'opposite brained' but it's the first thing that I think about when making something, sometimes at the expensive of the algorithm.
There's a bird in the back of my mind literally every time I use the shell chirping at me to translate them all into something consistent, and then make actually useful manpages for them.
The world is complicated, we have too much to learn, communications and ramp-up are essential and part of the product even if there's genius under the hood. (I'm also looking at you Rust, Git).
The same goes for systems derived from BSD - in fact, I wonder if some of the differences between MacOS and Linux comes from the former's origins in Mach/BSD via Nextstep.
My understanding was that this is most of it. The problem is relatively basic—macOS uses the BSD versions of most unix tools, which means basic commands like sed sometimes function quite differently!
Rsync’s man page opens with examples, and even though I never actually those specific commands, they are usually enough for me to remember how to do whatever I wanted to do.
It helped a lot that 15+ years ago I committed to running desktop Linux as a daily driver. Especially back then when things were much rougher on the desktop, using Linux as a daily driver means occasionally doing something in the shell.
Ultimately, just like learning a programming language, one must have a practical reason, a project, to make it worth the while to learn as you go. For someone interested in learning to use the power of the shell, don't just go read a book or do some exercises or tape a cheat sheet to the wall. Instead, commit to using it for daily tasks for a month, or for maintaining a project entirely in the shell, not even using a GUI file manager. It's tough at first but so worth it!
One thing the author didn't cover is how you can share reified knowledge when you write shell scripts. (It's the same with other programming languages, but they aren't as easy to write or modify.) That is really really powerful, because you can not only share it with others but also with your future self.
I wrote about this more here: https://letterstoanewdeveloper.com/2019/02/04/learn-the-comm...
I INVERT this, and write a shell script with comments :) That way someone else can reuse your knowledge more easily (and your future self as well).
Examples:
Constructing a big curl command to use the Zulip API, and also using jq for the first time. I used this to easily make a blog post out of a long Zulip thread [1]
https://github.com/oilshell/oil/blob/master/services/zulip.s... (oops some tabs snuck in here)
Figuring out how to use uftrace (and successfully optimizing Oil with it [2]):
https://github.com/oilshell/oil/blob/master/benchmarks/uftra...
Though one issue is that shell scripts don't really specify their environment, but there is a large number of tools growing around containers that can solve this problem. (Basically Docker is being refactored into something more sane; thank you to OCI and others.)
So I hope to integrate the Oil shell and containers more so shell scripts are more reproducible. I mean most of the container tools are already command line tools so in some sense it's already done, but you can imagine closer integration (e.g. not just passing code as text from outside the container to inside the container).
----
I wrote some notes about the documentation issue here: http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#...
And one thing I've wondered is if Oil should literally run shell out of markdown, so you can create executable docs. I can see it being useful, but it might be something you should do with a separate tool that converts markdown code blocks to a shell script...
[1] http://www.oilshell.org/blog/2020/11/more-syntax.html
[2] http://www.oilshell.org/blog/2020/01/parser-benchmarks.html
Writing that knowledge in some document is nice but in reality it's just more effort for no real gain, especially as it only helps if the relevant note can be found quicker than an online example/explanation.
To this day, I've still never quite gotten the hang of the whole "trap" mechanic, though...
I agree that IDEs and the tools that accompany them are powerful and can make you more productive. Learning them is an investment that can pay dividends. On the other hand, most of the IDEs I took the time to learn in school are now obsolete. I still use the shell today because those skills are still relevant and have gotten me out of a jam plenty of times. I've chosen to prioritize learning tools that are reliable and lasting even if it costs me some productivity.
I don't think that a shell should be a complete programming language. If you need a programming language, then better use one. There is xonsh if you are looking for something like this.
I think there should be a better bash with an very clean and consistent interface. Some of the most commonly used utils like awk '{ print $2}', sed, grep, sort, ... should be included out of the box to improve performance and to provide consistency across distributions.
I am not the author of that piece, but the analysis is wonderful.
What do you mean exactly? If you have pipes and lists (e.g., the lines on a file) you are Turing-complete and people can and will make arbitrary programs. How would you like, precisely, to restrict the shell language so that it is not Turing-complete?
Bash (and other shells too) is like it is, because it's an accretion of 50 years of history rather than a singular top down design.
No, it wouldn't; you'd just get EINVAL if you pass a filename containing "\x20" to open or other syscalls. If a application wants to use "\xC2\xA0" in a filename, it can do that, or it can not do that, same as currently. Same applies if it wants to use, say, "\xAA\xFF".
PowerShell; although you have to use long form to do it:
PS C:\> ${var with spaces} = 1
PS C:\> Get-Variable *space*
Name Value
---- -----
var with spaces 1
PS C:\> ${var with spaces} + 2
3In SQL, notably. Such identifiers need to be quoted, of course.
ISTR that many dynamic languages allow this with dynamic assignment features, but you often need to use similar dynamic access functiona rather than simple variable access to access them.
Ruby also allows whitespace if it is unicode but not ASCII whitespace, without use of dynamic assignment/access methods.
SQL allows spaces in object names but requires quoting, similar to shell.
My dream OS would forbid path names that don't match \w in grep.
You can have newlines in filenames, but you can also pass binary data with \0 if you want (it's 8 bit clean). Or you can use read -0 for a NUL delimiter, and eventually length-prefixed blobs with netstrings.
The way I now think of it is that Oil should have all 3 solutions to "the framing problem" in networking: delimiting, escaping, and length prefixing. So you can convert from one regime to the other by using shell.
People joke about Emacs being an OS but it’s really just a different kind of shell, oriented towards text editing instead of issuing commands.
I am not sure if we should care about the differentiation between TECO-Emacs on one hand and Elisp-Emacs on the other.
Perl had very good performance, despite being a "scripting" language, because so many of the tasks that a typical shell script might need to do were supported by built in features in the language (for example handling regular expressions, interacting with command line arguments, exec'ing other programs).
Today, once a shell script requires very much logic, I'm inclined to use Python. I know Python and find programming in it easier than Perl and bash and other shells. Furthermore, other programmers understand Python better than bash.
If these design points turn out to be over-constrained and have to make major compromises, yes the next thing I would try would be a small separate language for shell.
So far nobody has complained about any of this, I'm guessing because it looks very familiar, and that was intentional. Oil takes some pains to literally look like shell + Python syntactically, with better semantics.
I noticed that a few other shells are having problems with this command/expression distinction, and I discussed it like 3-4 years ago with Ilya Sher (of NGS) and a few other people. They were also having the same problem.
For example, does / mean a path separator or the division operator? Does * mean a glob or a multiplication? In Oil, this is no problem. It's obvious depending on the context.
Though I'm interested in more feedback on this, and the latest release is available to try as always :) https://www.oilshell.org/release/latest/
---
The awk and make integration is still doable but not done. Awk might require a notion of "lazy expressions" or "lazy arg lists", which would be shared with dplyr-like functionality. Oil's Zulip is open for discussion on these ideas :)
[1] https://www.oilshell.org/release/0.8.5/doc/command-vs-expres...
Absolutely agreed, shell is in many respects terrible.
>I don't think that a shell should be a complete programming language.
On the contrary, I think shell should be a more complete programming language! Drop the stringly typing and add actual types (hence eliminating 80% of bothersome awksedgrep magic; yes, no need to tell me it's a real tall order), add proper error handling...
I truly value simplicity where possible, but I think that the primary interface that I use to communicate with my computer should be as powerful as possible. Numerous times I've built up a shell pipeline only to realize near the end that I need to do something that shell is horrible at, and had to redo the whole thing in Python from scratch. I cannot really see a reason to limit it.
>Some of the most commonly used utils like awk '{ print $2}', sed, grep, sort, ... should be included out of the box to improve performance and to provide consistency across distributions.
UNIX "philosophy" got a lot of things wrong, but not this one. These common utilities you mention are good especially when they're separate, because it's not the shell's job to improve performance, provide consistency, etc... of a dozen and a half different utilities, a number sure only to grow in time as people discover what commonly used thing they want in their shell. Though I'll give you that `awk '{ print $2 }'` should definitely be its own utility or a built-in.
Which is also one of the reasons I prefer using Vim over Emacs; I can embed Vim in any terminal multiplexer of my choice and combine it with various terminal programs. Every single part of that environment is easy to understand and easy to replace with another program or script. It's less homogenous and consistent, but I am fine with that tradeoff. As one might guess I am not a big fan of IDEs.
For example, find all the .py files under the current directory, find the ones that have changed in the last day, and then print the lines (with their filenames) that define classes:
ls -fr \
| select (f: f.suffix == '.py' and now() - f.mtime < days(1)) \
| read -l \
| select (path, line: line.startswith('class '))
(Yes, this is doable with find and awk, but this is just an accessible example that gets across the idea of Python functions on the command line.)
from marcel.api import *
for path, line in (ls(file=True, recursive=True) |
select(lambda f: f.suffix == '.py' and
now() - f.mtime < days(1)) |
read(label=True) |
select(lambda path, line: line.startswith('class '))):
print(f'({path}: {line})
read(label=True)You don't even need awk for that one:
grep ^class `find . -mtime -1 | grep py$`
read -c foo.csv | sort (*x: int(x[3]) + int(x[6]))
Partly, I was also scratching an itch: I prefer using Python to exploring the sublanguages of various shell commands. Much cleaner, in my opinion.
Yes, that's a much better example. Here the "obvious" shell solution involves using awk to compute the sum of the two columns and putting it as the first field, sorting by the first field, and then removing that field. I guess a "rosetta stone" of such examples (e.g. on the frontpage of the site) would make a strong case for the interest of marcel.
awk -F, -v OFS=, {print $2+$3, $1, $2, $3}' foo.csv | sort -n | cut -d, -f2- awk -F, -v OFS=, '{print $4+$7, $0}' foo.csv | sort -n -k1 -t, | cut -f2- -d,
awk -F, -v OFS=, sets the input and output column separator to comma, '{print $4+$7, $0}' outputs the sum of column 4 and 7 before the rest of the line.
sort -n -k1 -t, sorts the file numerically on column 1, with comma separator.
cut -f2- -d, removes column 1, with comma separator.
This is of course not robust for general CSV files, but I don't think OPs marcel is either. A robust solution requires a proper CSV parser.
The biggest warts I see in the classic unix solution is that all the tools use different flags for the field separator.
edit: if you know that the csv doesn't contain tabs, you can omit some flags for a more concise
awk -F, -v OFS='\t' '{print $4+$7,$0}' foo.csv | sort -n -k1 | cut -f2-
<foo.tsv awk '{print $4+$7,$0}' | sort -n -k1 | cut -f2-I haven't tested corner cases, but marcel relies on the python csv module, which is probably better than any initial attempt at a parser that I could write in an hour.
This is what I meant about sublanguages. Many people, (myself included), would need to go to the man pages to find the necessary arguments to awk, sort, and cut. I find it much easier to just write a little Python, even if the end result involves more typing.
[1] https://code.joejag.com/2020/a-month-with-powershell.html
It's a shell embedded in the Racket programming language. If you're familiar with Xonsh, it's similar, but both more powerful and less polished. It allows easy, recursive mixing of shell code (not posix compatible, but with a similar feel) and Racket. Its interactive mode is not polished (I need to write a better line editor), but it works, and it's great for programming.
I haven't worked on it much lately (I need to wrap up other things to finish my PhD), but this weekend I added support for user-programmable substitutions with automatic cleanup. As an example, beyond common substitutions like process substitution (IE <() and >() in bash), I added a demo “closure substitution” form that allows you to send Racket functions to `find -exec`. Importantly, this is something that a user could add, because Rash is super extensible and malleable.
Shell is a great DSL, but there are huge advantages to having an embedded DSL rather than a stand-alone DSL. Rash inherits all the cool features of Racket and can be mixed with other Racket languages (eg. Typed Racket, Honu, etc), has advanced features like first-class delimited continuations and the world's most advanced macro system (which makes Rash possible), and any shell script can import functions from Racket's catalog of third-party packages. Embedded shells like Rash allow a shell script to be copied from interactions like you do with Bash, but then grow past the “rewrite in Python” stage gradually with no rewrite -- just a gradual transition from being more shell code to being more “normal” code.
Though I'm kicking myself for continually not getting around to rewriting all the documentation, which is still very poor.
Marcel is also somewhat verbose. It exposes Python functions on the command line, so when you need to write such a function, it starts getting long. E.g. if you want to sort files by last time modified:
ls | sort (f: f.mtime)
bytime = [sort (f: f.mtime)]
ls | bytime
Absolutely! Some work should be put into making it way more intuitive. Having to remember what every flag means (which is different in every app!) is not intuitive at all.
By default there should be some sort of intelisense auto-complete which can also provide guidance on what on earth all the flags mean, and maybe eventually it could tell you what it expects to do before you run the command. For most users otherwise it ends up being "paste in this command you googled with a load of random flags you don't understand and won't remember"
Why does terminal have to look like terminal? Like why is the 1980's ASCII telnet style the only way to do this?
The ASCII fish image on boot and fully ASCII menus only reinforces again that this isn't a modern interface, its an improved 1980's interface.
Why, for example, can autocomplete not look like this? https://code.visualstudio.com/docs/editor/intellisense
Why do progress bars when you are downloading from pip not look something like this? (still in-line with the terminal like a sparkline) https://docs.microsoft.com/en-us/windows/win32/uxguide/progr...
Like surely everything doesn't have to be SO 1980's if we want shell to stay relevant.
for real though, after looking at your examples I feel that features like those would fall on your terminal emulator to bring to the table, not the shell itself.
specifically, I don't want to imagine trying to connect to a headless server during some crisis over bad wifi and have the shell think it needs to send a bunch of graphics to me.
Four Features That Justify a New Unix Shell http://www.oilshell.org/blog/2020/10/osh-features.html
$ git status -s | grep '^ D' | awk '{ print $2 }' | xargs git checkout --
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
$ git status -s
D "g h i"
Haven't tested it, but you may also need to throw a "tr '\n' '\0'" in there and call xargs with "-0" to make it happy.
This will still break on files with a CR in the filename.
(CRs are allowed in unix filenames but, imho, they should not be.)
For a one-off thing like the one mentioned in the article I'd say it's almost ok, although I'd put `git checkout --` in the category of risky commands if you don't have any backup.
git status -s | awk -vORS="\0" 'gsub(/^ D /,"")' | xargs -0 git checkout --
Anyway, I suspect the following is POSIX compliant:
git status -s | awk 'gsub(/^ M /,"")' | tr \\0 \\n | ... git status -s | awk 'gsub(/^ D /,"")' | tr \\n \\0 | ...I mostly hit things like these with alpine containers and openwrt. OpenWRT I guess could be considered "fringe operating system", but alpine (at least for containers) seems reasonably mainstream?
When it comes to examples like that posted by the OP, I like the combination of piping/pasting to mate and multi-caret editing for most scenarios where others would reach for awk/xargs.
Here's me following the same example scenario but with multi-caret editing (slowed down slightly):
https://user-images.githubusercontent.com/226503/101993951-6...
Step by step:
1. git status -s | mate
2. select " D" with arrow keys + shift
3. command-E macos default for "use selection to find"
4. option-command-F to find all (multi-caret editing starts)
5. type "git checkout" to replace " D"
6. command-left to move cursor to start of line, then shift-command-right to select to end of line.
7. copy
8. select all + delete (clear document)
9. paste,
10. press return to insert newlines
11. select all + copy
12. switch back to terminal
13. paste
Other tools and languages come and go, but the shell and SQL just keep on providing value.
I've got a bunch of SQL resources I've collected on my blog that you might find helpful: https://simonwillison.net/tags/sql/
Best hope your file names don’t have spaces in them.
One of the downsides of the shell on Unix is that everything is usually text meant for humans so you have to spend an annoying amount of time dealing with delimiters that make it easy on the eyes but a pain in the ass.
learning to be comfy in shell is still worth it though. It's saved me, at minimum, hundreds of hours of work.
As an alternative, try Sublime with multiple selection (or other editors). With multiple selection skills you can transform your lines in a WYSIWYG interactive format which is much easier to work with for me.
(To use it well you need to know the following shortcuts: split selection to lines, select next, and the alt+arrows jumps to next/previous word)
This won’t mean you don’t need to learn how to use shells, but is still pretty fun to use.
- Use "head" to reduce your data first and then quickly iterate on the pipeline. (Or sample [1] if head isn't representative; this is rare in practice)
- Use tmux (or at least 2 terminals) with shell. The left side should have your editor, and the right side should have your shell (just like the screenshot in the blog post -- that's what mine looks like too)
There is definitely a place for GUIs (IntelliJ CLion beats the GDB console for me) but you can get really far with vim, tmux, and shell.
https://unix.stackexchange.com/questions/108581/how-to-rando...
Yes! This saves me time consistently. I could wrangle a shell script, or where pattern-matching is essential, using text editor to select w/ keyboard shortcuts is VERY powerful.
I've been using editors that are language aware since at least the late 90s. Depending on the language they'll show me all references, take me to the definition or declaration, stack those jumps so as I follow the links I can pop back a level to where I was. All of this is instant. 1000x faster than opening a shell and trying to grep all the project files for word phrases which, not being language aware, can't tell if one 'foo' is relevant or irrelevant from another 'foo'. They'll also let me refactor various things from renaming a method/class/variable/function and correctly fixing all the related files to doing things like changing class members to getter/setters and other things.
The difference is like using a hammer vs using a nail gun. There are times where they hammer is useful but given the job of constructing a building a nail gun will get the job done much faster.
For example, if you name a member "flag" and you have several types with the same member, your IDE can tell you where this flag is used, so it's not a big deal if you wanted to refactor or are trying to track down a bug. But god help you if you're looking at your code in a diff viewer or in your browser in an online repo.
In my experience it's not common to name members/properties taking other classes into account so I might have a Window class with "visible" and Shape class also with "visible" and Player class with "visible". I don't think I've ever seen a project where they made those "someWindow.windowVisible", "someShape.shapeVisible", "sompePlayer.playerVisible" just for the sake of searching regardless of if they person is using an IDE that could tell the difference or just notepad/nano.
All of my experience is they would all be just "someWindow.visible", "someShape.visible", "somePlayer.visible". In other words, that choice has nothing to do with IDE or no IDE.
Is there some other example you were thinking of?
Still, a lot of people start out programming with complex IDEs and end up being unable to run their code without the "play" button of their IDE.
Starting out, or only ever leaning an IDE is also hiding a lot of things from you.
For me, knowing the shell isn't about IDE vs. shell tooling, it's about, whatever you use, be aware and knowledgeable about the foundation and being able to do stuff even if there is no IDE. An IDE is a great tool, but it shouldn't be a crutch.
And vim without extensions - I'm pretty sure that's a rare thing among programmers.
Yep. Another thing I hate is IDEs that build their own project files that tie you to them. A good IDE works with standard tooling, not as a replacement.
The winds shift, and the ship of progress sails a direction that is difficult to chart, but vim....vim never changes, and for that I am very glad.
A lot of people start out with shells and end up being unable to understand how to construct a switch or adder in logic gates. A shell is a great tool but shouldn't be a crutch.
At the same time, I know that's extremely unreasonable to expect. I guess for the shell and editor, you can argue you ought to understand it stripped of abstractions, since it's actually the level you're working at, while most people don't work at the logic gate level.
It starts at boolean logic and works up to a web browser and then some other stuff.
Edit: It's a high level covering of each topic of course, it does have to fit in a book. It's ~444 pages.
It doesn't go all the way down to the physics of semiconductors, but it goes down to logic gates and all the way up to tetris (there's a related course/website called nand to tetris: https://www.nand2tetris.org/)
If you want to go further down, check out Jeri Ellsworth's old videos where she built a transistor at home: https://www.youtube.com/watch?v=w_znRopGtbE
However, that's exactly the path every Electrical & Electronics Engineer classically took.
Most people in my class ended up at the top of the stack (operating systems and software engineering) since the jobs are more numerous and the pay is better.
The usefulness of knowing the whole stack has diminishing returns though.
It can be helpful in a large company where I can make connections that others can’t, but in a small company it’s less relevant unless I were solely focused on one specific connection (say, how machine learning using math as designed in an ALU inside a CPU or GPU is ludicrous—go analog and get four orders of magnitude speed up, all that time waiting for carry bit propagation, yuck!)
Your time is probably better spent learning to be a good generalist or specialist in your field, rather than knowing inside so many layers of abstraction.
cf. https://www.pbs.org/show/crash-course-computer-science/
No, it won't give you the level of understanding that an actual comp sci/engineering degree will (after all, it's just a couple dozen ~10 minute videos), but it does touch on all the major concepts and through the series builds on concepts presented earlier.
Perfect world, you know everything.
In reality, you can't and don't need to. Knowing some shell stuff will be useful for most if not all programmers.
I think my chances of ever understanding what Symbiflow does are quite small, but that is a shortcoming on my side :D
And even if I do, I have to learn semiconductor fabrication next!
We could also make the opposite argument for any level of abstraction: can you really say that someone who buys apps for their iPhone and runs them is less of a programmer than someone who actually programs? If programming is just getting a job done, and being a good programmer is just picking the right tool to get the job done...
The real question is pragmatic, not theoretical. Does your dependence on a tool sometimes make easy things impossible or encourage misunderstandings about lower level processes that lead to bugs or inefficiencies? Is it simply too big or expensive to run in all of the places you might want to program? Does the tool make up for that lack of flexibility with increased productivity? Those are real questions that you can ask about any specific tool (including the shell.) It doesn't mean anything to ask them about tools in general, and the idea that sacrifices and benefits must all come out even in the end is just the law of averages.
Starting out, or only ever leaning a compiler is also hiding a lot of things from you.
Now, I am familiar with assembly, I live and breathe it honestly, but I don't gatekeep and say "you're not a real programmer unless you've written your own assembler macros". Some people are happy in their IDEs and see no reason to move to the lower-level of abstraction in their tooling. I don't see why we should fault or shame them for that.
Personally, I see shell as a poorly-designed, very error-prone language and I think we should move past it, onto far better tools.
I'm talking about the befits of knowing more stuff.
I'm not arguing against IDEs, I'm only arguing about the benefit of also knowing other stuff.
Better tools: I'm very interested in, what I would in lieu of a better term describe as, Bret Victor's ideas and the example of Swift playground. At the same time, I'm very sceptic against everything that isn't a simple, human readable Textfile underneath. One thing I keep thinking about is making diagrams executable, but I haven't seen a graphical programming language that convinced me. Still, I think this could happen some day, if someone finds the right design.
I recently worked with a colleague who couldn't run the python program he wrote without the IDE. The IDE was not installed on the PC we were testing the program on.
He is a better programmer than me.
And I'm not saying "people who are using IDEs don't know anything".
I'm literally only talking about the benefits of also knowing some basic shell and CLI stuff.
edit: It's so strange to not only demand the right to not know how to do something without a complicated tool (a right which is inalienable and not threatened), but to also demand that people who do know how to do that thing not think that it is better to know how to do that thing. I get that when one has invested their time into a tool they want to defend its usage, but people who aren't dependent on a particular vendor's tool have also invested time, and should be able to think it's better to not be dependent on a particular vendor without it being considered violence or bullying (in the modern twitter sense.)
Personally I've found a diverse set of viewpoints to be incredibly valuable on the teams I've been a part of. I have my own biases on testing, stability, performance and seeing how others approach it has broadened my understanding of how to build software.
To put it another way, the parent could have said "IDEs are awesome and I've found when you combine that with an understanding of the CLI you have an awesome set of tools at your disposal that are greater than the sum of their parts" instead of making it an exclusive trade off or implying that IDEs are limiting.
But that would be a completely different argument. Their whole point is that there /is/ an exclusive trade-off, which is different from what they're trying to say.
If you want my opinion, they both piss me off, just in different ways. The IDE requires me to use my mouse, even with the best Vim impersonation plugins. I hate that, but I put up with it because the autocomplete, syntax completion and indexing just works so much better than the Vim equivalents which fall apart under heavily load. And conversely, trying to edit remotely complex projects is a nightmare with Vim, no matter which distributions I use.
Which maybe is a different tone of saying exactly what you are saying.
So not sure what your point is.
At the risk of sounding patronizing as much as we'd like to everything to be binary pass/fail and survive on the technical merits or semantic details, how you frame things and driving communication in an inclusive way is important of you want to convince people that something is worthwhile.
You can read my comments as gatekeeping or as the exact opposite - an invitation to come through the gate and see what you can learn here and take what is useful to you.
True but you're also at the mercy of whatever company is responsible for developing those IDEs. I'm not advocating everyone do development in shell but I feel like you should know what's going under the hood when things go wrong.
With all the IDE companies being so ruthless, I wonder where are all those screwed-over IDE users running to.
I will reply with a car analogy. You can be a professional driver even if you don't know how to write the chemical formula for the combustion in your motor. You can not be a good professional driver if you don't know how to change a flat tyre or check your oil. No matter how good your driving is, if you cannot change the oil you are an incompetent driver. Maybe in the future, cars will not need oil. So good, but today they do and you need to deal with it.
Likewise, today's computer systems rely on the shell. If you are shell-illiterate, you are an incompetent programmer, no matter how good your programs are.
That vim and bash and all of these terminal-driven ways of developing are the unique privilege (and I do say that unironically) of working with *nix systems.
That’s not to say Windows devs don’t use vim, but in my (limited) experience, it’s about as common as using a gas-powered generator to charge your Tesla.
My comment is more addressing the article where in the author mentions vim with no extensions and grepping. IMO that's often the wrong tool for the job (writing code). It can be a useful tool but there are often better tools for that job. Knowing when to use one vs the other and therefore knowing both is very useful.
Tools can be useful however, such as the hot reload of flutter/dart. That's the kind of improvement I really appreciate.
Every tool, plugin or functionality takes up some space in my mind. Once I know the tool really well, that space becomes smaller and the benefit of having the tool outweighs the cost. This is why I only add tools and plugins slowly. One at a time, only after I mastered it I add something new. And I weed out my tools from time to time and remove things I don't use often enough.
Maybe the people who use feature rich IDEs just have more RAM and a more parallel brain, and the CLI people have serial brains :D
But it's worth pointing out that IDEs provide depth for working on a given project. If you want to provide breadth across many projects (like writing a custom linting script), you do want to be she'll literate. Not every automation task can be specified in with grep and such, but quite a number of them can. And the ones that can't be written in terms of the shell often benefit from hybrid solutions that combine shell tools and specialty tools like parsers.
Not commenting on who is right or wrong, but there is a key difference in your workflow here. OP is not opening a shell at this point. The shell is the first thing being launched, vim is secondary.
It'd literally be click status column to sort, click top entry, shift-click bottom deleted file, right-click, restore. Done, a couple of seconds without even thinking about it.
And somehow this simple task in a GUI inspired a blog post about how useful the shell is.
I'm not going to deny a shell is extremely useful, and knowing far more than I do makes you a better programmer, but there's also using the right tool for the job.
This was originally done in a couple of seconds without deeply thinking about it (hence originally using grep instead of `awk '/re/'`).
(Edit: quote)
1. Tables you can sort
2. Tables you can sort on the thing you actually care about
3. Being able to select multiple entries and act on your selection
It’s not too hard to imagine criteria which could make the problem much harder for a gui, e.g. only doing the operation in a subdirectory (if the sorting is stable this isn’t so bad), or only on files not in a subdirectory (stable sorting won’t help here, you’d need the gui to let you eg search for / and then remove those things from your current selection), or only operating on .h files (you’d need a search->select feature again), or doing an operation that the gui doesn’t support on multiple files, or passing the list to some other tool (maybe you want to search makefiles for references to the deleted files).
Something that a gui may be able to handle but that often screws up shells would be file names with newlines or spaces in them.
When's the last time you accidentally delete 200 files and needed to restore them? Probably because you were using a shell...
Oh yeah, and if you accidentally delete folder in a GUI? Usually you can just press ctrl-z, undo!
In any case there are like a bazillion of GUI git tools. Are you certain each and all of them support this feature? If someone doesn't know any of them, what are the odds the first GUI tool they install has this feature? The hidden cost of finding the "right" tool is overlooked in its advocacy.
With the shell, the second time you do it is a few keystrokes, if even. Wanna do it across multiple projects, it’s super easy without ever loading any of them in your IDE.
Wanna share it with teammates, you simply need to pass a script file to them, and they can do it without having to do anything.
And now finally if you need to incorporate this behavior in your headless CI? It’s the same script, instead of having to putz about with the CI configurations or whatever.
b) a few keystrokes === a few clicks
I feel like this metaphor falls short.
The IDE, IMHO, is more like a multi-tool with pre-chosen use cases, but everything fits together nicely.
Using the shell is more like choosing the exact tools for the job, and maybe connecting them together in a make-shift fashion.
Usually, its super convenient to have all those tools integrated because you can scrape some paint and pound a nail and pull a nail all right then and there with needing to do any swapping out or hunting for the right tool.
But the instant you want to be able to hold a bolt in place with the wrench function while simultaneously using the hammer function, you're shit out of luck. The painter's tool, on its own, can't be de-composed to do those things simultaneously. Plus, it's not as good at being a general purpose hammer or a general purpose wrench. But in the context of the tool, you don't really need a full fledged hammer or wrench. Having a single small tool you can hold in one hand is usually more pragmatic, even if less versatile.
Probably still gonna want the toolbox to be available for when you need it, though.
I’ve never seen advocacy for one over the other as a competition. I do plenty of both types of editing, and often edit the same files simultaneously with three different tools.
(Example: Editing notes in Notable, Atom, and vim.)
Know any C/C++ IDE that would do this? I transitioned from Qt Creator to VSCode and none of those have helped me with this cool nicety you mention.
EDIT: I should add that for me, exploring foreign code bases has been hugely improved since I discovered Sourcetrail, so I'd suggest having a look at it to anyone interested in code exploration tools: https://www.sourcetrail.com/
I do use Jet Brains other editors and Intellij has flawless Java code navigation (among other amazing features). Further, I've found PyCharm provides Python code navigation that is about as good as possible for such a dynamic language. Therefore I'd expect CLion to provide top notch code navigation for C/C++.
https://stackoverflow.com/questions/35424367/how-to-navigate...
I do this in stock vim using ctags files. You can generate them with anything, though “exuberant ctags” is very popular. Ctrl+] when over an identifier jumps to its definition, and there’s a tag stack to jump backwards similarly (you can also use the jump list). This too, is instant. It’s also an essential feature of the help system in vim for navigating cross-references.
There’s also other robust IDE-like features built-in. Syntax-aware code folding (with adjustable fold levels, see ‘folding’), compiler integration so that it’ll highlight where compilation errors occurred (see :make, makeprg, errorfmt) and so forth. Not saying your life won’t be better without some plugins, but stock vim is quite a bit more powerful than having to shell out and grep.
Other than that, a good deal of open mindedness and skillful use of IDEs is not a bad trait. Let's not be extremists.
You are not back to negative one, more like you are at square 85. There is nothing stopping a developer from supplementing something that the IDE doesn't easily support. For example I use PyCharm these days and I when I make a change to a file I automatically want the test associated with that file to run- this is not supported in PyCharm (or atleast I don't know how to get PyCharm to do it) so I wrote a script that runs in the background of my shell watching for changes on files on my project and running appropriate tests.
The point if an IDE doesn't have a quick way of doing something that is necessary for your workflow, it doesn't mean that the 20 other things that the IDE does are useless. Any working developer should continue to be able to extract value from the features that the IDE provides and jump back to fill the holes themselves (via shell scripts or what have you) instead of saying "Whelp this means IDE is a crutch now and I must revert to only using 'Unix as IDE' and only relying on sed/awk/grep and friends" to aid my development workflow.
Example: I work with parallel filesystems quite a lot. Often I will need to parse through GB of logs spread across a cluster to track down a particular sequence of events for troubleshooting.
I could load up a fancy IDE with all the bells and whistles, one that instantly refactors all my 'foo', and still spend a (very) non-trivial amount of time carefully crafting (and documenting, and debugging) a proper application to solve the particular problem I'm working on.
OR - I could rattle off a pipeline using a few general tools and get the needed details in less than a minute.
You analogy about hammers is particularly amusing, invoking the old saw "If the only tool you have is a hammer, every problem looks like a nail". You take this to the absurd degree that you end up championing nail guns!
The OP is comparing IDE v.s. shell in terms of "exploring" and "editing" the code. In that case IDE is a powerful nail gun and championing it isn't wrong.
It blew my mind just how powerful the IDE was and how it massively increased my productivity. Putting aside code completion and code generate, I found it much easier to navigate a codebase using a proper IDE that fully understood the language. When I contrast that to my previous solution, a highly customized emacs configuration with lots of packages and my own functions, the two just don't compare.
IDEs for the most part are not covering for language flaws, they are helping developers be more productive by removing the need to focus less interesting parts of the process and focus on the the problem being solved. While the need for an IDE is more important in some languages over others, there are a set of common problems that exist in every language that the IDE can help solve (navigating the codebase by jumping around to symbol definition/usages, auto importing packages/modues, intellisense, refactoring).
Refactoring does not mean going back and refactoring 6 weeks worth of work that you seem to have in mind. Most people who refer to refactoring are referring to it in the context of multiple small refactors during very short development cycles (every few minutes or hours). My workflow and the workflow of many developers I have worked with goes like this:
* Write some code (not more than 200-500 lines)
* Write tests
* Refactor if needed- potentially rename variables, potentially extract some code into methods, potentially pull stuff up into constants, potentially introduce a new class or interface, potentially change types of certain members, potentially visibility of certain members or methods.
* Repeat for the entire workday when you are not in meetings.
Refactoring- specifically the kind of small scale refactoring that you do as go through your workday to me is almost as important a part of the development process as anything else and IDEs remove almost all friction from it by removing any cognitive load of the task from the developer.
Maybe without the IDE, the code would have ended up a little cleaner.
When you can’t easily jump from file to file and get intellisense like completion you are forced to design modules and data models which can fit inside your head. I imagine this can be a feature in some cases and a limitation in others.
I tend to only use an IDE when refactoring, and a text editor for everything else. So far I haven’t had to write or work on code which I can’t keep all in my head; maybe I have just been lucky with the codebases I have worked on?
Technically true, but it's only a limitation in the sense that not being allowed to use asbestos shingles and lead pipes is a limitation for someone building a house.
> 1000x faster than opening a shell
Don't close the shell.
> trying to grep all the project files for word phrases which, not being language aware, can't tell if one 'foo' is relevant or irrelevant from another 'foo'.
IntelliJ is the gold standard according to my Java buddies and when I search for a class (by hitting ctrl-N ... go figure) I frequently don't get any results. Some kind of misconfiguration will silently make things not work. It happens enough that I have to second-guess its results, even when they're correct. Now I only use ctrl-N to jump back and forth between classes I know exist. Actual searches I'll leave to find/grep/ag.
Likewise when I optimistically right-click some part of my project and click 'Run all tests'... "No tests were found".
It's fine to try to open the random project in your favorite IDE and see if it just works. If it doesn't, and you just need to figure out what the code does, fix a few lines of code, going back to a text editor, grepping and trusting the CI to run the tests itself is just less overhead.
If you work on the same project for a longer period of time, you can easily justify the upfront cost of configuring your environment properly (and have your class searches and 'Run all tests' running).
I've also used intelij idea with Java, and while it was much smoother than anything in the typescript ecosystem, it's definitely not faster than using grep.
Vim can do this too and (rip)grepping the code is only needed if you're looking for, you know, text. I have 'gd' mapped to "go to declaration" and 'gr' to "find references". Works across dozens of languages, can be further scripted upon. Basic refactoring (renaming class method in several files) should also work, but I use it so rarely that actually can't testify about it (I use rope bindings for python refactoring but it's usually limited to a single file).
It's not as honed as purposely built 5GB IDE of course. But having one editor for all languages and syntaxes is really nice. Things like VSCode have this one too, but I wouldn't call them and IDE either.
I haven't been using IDEs any meaningful amount of time as I'm not a developer, but if the things you mentioned (go to definition, changing names) are the main reasons for an IDE, one could argue that terminal workflow has substantial benefits, like better understanding of the toolchain, simpler integration with ci/cd tools, endless customization, single tool across languages and other standard cons of CLI tools.
I do, however, find vim without extensions useless for editing code as well.
If you dislike POSIX sh, consider looking at Plan 9's rc before you look anywhere else (including bash): http://man.9front.org/1/rc
Also, I would advise you to learn about these four tools in depth: sh, sed, awk, and make. Learn when to use each, and don't use one when another would be better.
The "make" tool is really powerful, besides its use for calling compilers. You can write really nifty scripts with make that have automatically optimal parallel scheduling.
I immediately remember how great are Jetbrains IDEs. I just can't imagine how someone could refactor code with such accuracy and without hurdle just using Linux vi or any cli tool...
With multiple contexts, and sometimes complex behavior, leveraging shell seems key to debug and gluing things together.
And I think it's growing beyond CI/CD -- that is, Github actions is a more general platform than Travis CI, etc.
Good example: https://lobste.rs/s/oeelem/using_github_issues_as_hugo_front...
My pet peeve is that YAML is a terrible syntax for a shell script, i.e. there are if statements in a weird YAML DSL in addition to if statements in shell (which people already complain is hard to remember)
People complain about shell syntax, but YAML is even worse.
So I would like to make Oil a better language for specifying these "distributed shell scripts":
I think there are probably very elegant ways to combine yaml and shell scripting, maybe by careful factoring of things into their own actions. I hope so.
I will check out oil.
Since JSON is valid YAML, it's possible to generate JSON from an Oil configuration with embedded and parsed shell, rather than using YAML. (Actually there's no problem generating YAML either, but I suppose it's easier to think about.)
Example: I changed my own configs to use .yml.in and .yml and it worked on both Travis CI and sourcehut.
https://github.com/oilshell/oil/tree/master/.builds
Some brainstorming here on this feature:
https://github.com/oilshell/oil/wiki/Config-Dialect
The benefit would be that it would validate your syntax and possibly your schema locally, not remotely. Some services have command line tools, but I'm sure they don't validate your shell, only the configuration.
Another benefit is the ability to reuse code and configuration. For example, my .builds/ dir has a fair bit of redundancy in its configuration, and that could be "factored" with a real language. YAML has a limited << reference mechanism that does some of this.
I'm looking for feedback on Oil, as well as the YAML + shell replacement idea. I hope that Oil can be a flexible language that will let people write these sorts of front ends for many cloud services, i.e. "distributed shell scripts". And eventually enable portability across clouds too.
Maybe the recipient and the sender just compare hashes via a secure channel, but there's no mention of this.
I wish you would have at least mentioned it in your post that you were indeed relying on the security of the network and that nc by itself is not secure.
I'm not saying it would have been a real risk in your particular case, but giving these examples on a public website is basically saying "see, you can do it like this". I'm not saying you don't know better, but some of your readers might not, and may replicate your unsafe (without the underlying assumptions) code where the risk is higher.
This is especially bad given that there are safe alternatives with equal or better ergonomics.
Take for instance this video: https://www.youtube.com/watch?v=4djoOiLste0 The author shows how the output of "git status" is just a text, you can modify the text and add "git add" just in front of the files you care about and add the file to the index. There is no linear loop of command input that becomes command output, it's all one big buffer that can feed itself. Consider how there could be another window that perpetually shows the status thanks to a combination of inotify and git status, and you have a git view. Fiddle with the command line argument and you can choose wether to show whitespaces or not, whether to show a summary view or the full diff, or restrict the list to some files. Have another window where you can write some message, and a GitCommit command in the "Tag" of the window will use the whole buffer as a git commit message; boom, you have 60% of what I use git-cola for.
A bit more information can be seen by one of its creator here: https://www.youtube.com/watch?v=dP1xVpMPn8M. The possibilities are truly endless, and I haven't seen anything that resembles it. There is just no editor that embraces your platform the way Acme does.
No, Emacs is not the same, because Emacs doesn't integrate with your OS; Emacs is an OS unto itself. You can't really say it's integrated to the OS when everything is implemented in the language that only Emacs uses.
Thanks for the tip about Acme. I'm always looking for ideas about how to improve integration between the terminal based work flows and other applications, command line, GUI, or others.
Bash scripts are generally very short. If you need unit tests and a debugger you probably should be using something other than Bash. But saying you hate the shell because it's bad at something it was never designed for is silly.
Some commands are slow and caching their results can be a bit of a pain, if only because you need to use different syntax. Some commands shouldn’t be rerun as part of the cycle as they are side-effecting. Some commands can produce huge amounts of output which is annoying to deal with if you don’t think to run them into a pager. And sometimes certain kinds of data flow can be tricky to write (eg something like 1. List a bunch of things with a command, 2. For each thing pass it as an arg to a second command, 3. Print the result of that command next to the thing, where you want to write something like list-foo | tee /tmp/l | xargs -n1 query-foo | paste /tmp/l -, except this won’t work even if /tmp/l is a named pipe (it works if it’s a regular file and you add a | tac | tac after the tee)).
Finally, quoting is a bit of a horrific mess.
So here is a proposal for how I’d like a shell to work:
1. It still works as a shell so you can run ordinary commands.
2. By default all pipe operations don’t use a single Unix pipe but rather two pipes where data is transferred from one to the next by the shell (this can be done entirely in the kernel and very cheaply so long as you transfer in pages), and so you can see simple statistics (ie is anything happening), and optionally see some of the data going through
3. When you run a command, the output gets directed into something like a pager so it won’t make your history inaccessible, and furthermore, there’s a way to tack on a pipe into some other command on the end of your pipeline without it needing to rerun the earlier steps
4. Some commands like grep get interactive versions where you can see how the output varies live as you type in the arguments—there’s no need to keep readjusting your regex
5. Something a bit like awk or xargs as a first class feature of the shell language (and therefore some kind of notion of closure that builtin functions can use) to avoid some parts of quoting hell
Trying to do any non trivial scripting/programming in it will mess up your mind.
Oilshell perhaps?
# put interesting lines from history into a file, open an editor
function h2e() {
echo "#! /bin/bash" > "$3"
history | tail -n"$1" | egrep "$2" | sed -re 's/\s*[0-9]+\s+//' >> "$3"
"${EDITOR:-emacs}" "$3"
}
# prompt for h2e's arguments
function h2ei() {
read -p 'num. lines [1]: ' lines
read -p 'pattern [.]: ' pattern
read -p 'file [foo.sh]: ' fn
h2e "${lines:-1}" "${pattern:-.}" "${fn:-foo.sh}"
}
# fix the terminal settings for the duration of a command, revert afterward
function insane() {
save=$(stty -g)
stty sane
"$@"
stty $save
}
# bind F12 to our history grabber
bind -x '"\e[24~":insane h2ei'
Which would put the last 10 lines matching some pattern to the txt file?
In practice, I will do something, realize that the thing I just did over the last several minutes could be useful--or the core of something more general--and then strike F12 to start h2ei. At the prompts, I'll generally give it a nice round number like 10 or 20 or 50 lines, leave the default pattern, and set the filename to be something evocative in ~/bin. An editor will immediately open on my new shell script, and I can trim it down and continue hacking on it from there.
The key idea is that I get an easier path from doing something ad hoc to creating something more useful in the long run.
Many other commands have such an option. As an example, iproute2 has -json.
So... As long as we're talking about the shell, when will we be able to interact with sourcehut entirely from the command line? Right now I'm particularly looking at managing build secrets.
Being able to use it entirely from the command line is the only thing sourcehut lacks for it to become the best code/project/build management tool. At least for me anyway[1]; I'm aware that sourcehut's idiosyncrasies are not for everyone.
On that note, is anyone aware of any such platform that can be managed entirely from the command line? The one that comes closest is the github CLI, I think, and even that one still requires a browser flow for logging in and getting the token[2].
[1]: I am aware that I could dive into the code and maybe contribute this myself but a) there might be plans for this already; I believe there's an issue where it was discussed and b) I don't have nearly enough time to commit to it.
[2]: I know that this is very nitpicky of me, but I do truly want a tool like this; I think it'd be awesome.
https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/master/master-...
You can use it to, for example, do `ssh builds@builds.sr.ht submit < build.yml`
Expaning this to support secret management would be cool.
If you know the shell well and are well versed in a good CLI text editor such as Vim or Emacs, the programming language of the day doesn't matter. Whether you're coding or setting up some environment doesn't matter. Whether you're on your computer, connected to some server, or an embedded device doesn't matter.
EDIT: And eventually, you're not slower than someone using an IDE either.
http://facebook.github.io/PathPicker/
Pretty useful when paired with git. e.g.:
git status | fpp -c git addIf you put a bunch of location-prefixed lines in a vim buffer (either from running a command within vim or by piping to `vim -`) you can get them in your quickfix buffer with `:cbuffer!` and then step between locations with `:cnext` and `:cprev` (one of the few things my top-level .vimrc does is bind these to tab and shift-tab, respectively).
Mac Finder is my goto a lot because it's just powerful to see the tree. Could we blend the command line there?
And finally - for expert users this is not a problem, for for medium users, the fact that many of the commands have different flags in different orders is a pain. A brand new set of integrated commands - where everything followed a predictable standard would be great.
Also, given that I use emacs, I do lots with dired, and on odd occasions eshell. And I have gathered a bit of emacs lisp code as well.
Use keyboard shortcuts in your IDE, type commands into your shell. The killer idea that will make you more efficient is scripting (or creating macros in your IDE) for your own common tasks and assigning them a good name (or a shortcut in your IDE.)
I'm not going to disagree.
However...the presenter would be wise to spend a paragraph or two about actual benefits. In addition, tell us why I should listen to you. You don't persuade others with subjectivity and finger waving. Even something as simple as "...during my ten years as a software engineer..."
Otherwise, this is more rant than educational. Sorry, but I see this pattern far too often and am hoping to nudge someone here away from it.
gemini://drewdevault.com/
at the same time, it is a bit of a hack. what is the minimal non-hack version of this technique? seems abstractly like liking a file-descriptor.
thanks for the article. it reminds to get better at awk.
Using a "proper" media manager and adding the file to a play queue?
Rename the file 00WATCHME
But i have to say lately i dislike posts written in this command tone. Like "Do this to become X"/"Use this technology for Y"/"Don't do Z". Most of the time i end up feeling bad for not knowing something and therefore not being 10x programmer or even feeling antipathy towards the topic or author as i have different views. But in reality these things often don't matter or rather it can vary a lot person by person and context by context. You just don't need to be a shell literate to be a successful programmer. I wonder if more subtle introductions aren't better. Some which would spark interest but wouldn't leave a person with a bad feeling if he can't get into it right away.