Command line tools for productive programmers
earthly.dev
earthly.dev
The single best improvement to my command-line workflow I've done in the last few years has been switching shells. First I moved from Bash to ZSH, but about 2 years ago I switched again to Fish.
Why I love it: - history completion searches by default, so I can just type part of a command, hit arrow-up and look through only the relevant results (ZSH also did this, but Fish is better)
- tab completions are shown to you in advance (in light gray in front of your cursor)
- Basic configuration (which is good enough, honestly) can be done via a web browser. Just run "fish_config", and it starts a web server, pops open your browser and any changes you make there are saved to your actual config files. No more looking up syntax.
- aliases (which are called abbreviations) are actually expanded after typing, so the full command appears in your history. You can also edit the full command before running it, if you need.
What about bash one liners?
I had a bad experience with Ubuntu's (Debian's) dash shell many years ago and I've (lazily) just stuck with bash since then.
For me, yes I still write my scripts in hashbanged (ba)sh scripts; the only fish I write is my config.fish. Fish might be an objectively better language, I don't want to learn it and prefer keeping writing (ba)sh, as
1. (ba)sh is an evil I know...
2. ... that I must know anyway to work on servers (while fish's "utility" is low as used nowhere but around the fish shell)...
3. ... and shellcheck makes it (a bit) less of a footgun.
When I do need to write a bash one-liner, I just run it in bash :) . But it's a once-in-a-blue-moon occurrence, since pipes are the same, and fish added syntax for && and || a few years ago (before that, they didn't exist and you had to use and; and or;), and added support a few months ago for `FOO=bar ...` direct env.var setting (before that you had to `env FOO bar ...`). So, your typical `thing --blah | grep stuff | awk stuff | whatever && something` generally works in fish.
* do you have any trouble running shell scripts?
Nope. fish will run bash for most shebangs.
* What about bash one liners?
You will have to translate these.
You can still type "bash" to get bash.
Have you seen gron[1]? It can't do everything that jq can, but it is much more in the unix philosophy than jq. It simply flattens and unflattens json out into lines so you can use standard unix tools on it.
▶ gron
"https://api.github.com/repos/tomnomnom/gron/commits?per_page=1" | fgrep "commit.author"
json[0].commit.author = {};
json[0].commit.author.date = "2016-07-02T10:51:21Z";
json[0].commit.author.email = "mail@tomnomnom.com";
json[0].commit.author.name = "Tom Hudson";
▶ gron "https://api.github.com/repos/tomnomnom/gron/commits?
per_page=1" | fgrep "commit.author" | gron --ungron
[
{
"commit": {
"author": {
"date": "2016-07-02T10:51:21Z",
"email": "mail@tomnomnom.com",
"name": "Tom Hudson"
}
}
}
]
[1] https://github.com/tomnomnom/gronI am sorry for spamming this comment, but I wanted to share it with all the fzf users because I found it so game-changing:
I use "ESC-c" a lot. It calls fzf and fd, you type a part of your directory name, hit enter, and it cd's to that directory.
export FZF_ALT_C_COMMAND='fd --type directory'
That's in my .zshenv.It's incredibly useful if you have multi-layered project directories. Like ~/Projects/Terraform/Cool_Project - you'd hit esc-c, type cool, hit enter and you're in the place you wanted to be.
If I want to cd to let's say "~/work/john/some-project" I just type 'fcd' on the console and then "w j s" and "ENTER". Most of the times it work as expected and is really fast.
This is a mode provided by a helper that I wrote for fzf to make it easier to use and more useful. In essence it's just a shell script with a bunch of fzf tricks pre-configured.
when the transformation turns to be more complex than expected I can just copy and paste what I've made so far into a nodejs script.
you can also configure a .fxrc file to automatically import npm packages that you might find useful, shortcuts, or your personal functions.
entr lets you watch files and re-run a command any time they change. Whenever I'm working on a script, or go tests, or whatever test-like thing I'm doing that's not in its own bloated test harness, I reach for entr. Great software, does what it's supposed to every time.
Great project tho !
As an example, I use 'guard' [0] in a docker container, and on test failure it writes out to a file which is shared outside the container. It's easy for me to fire up a shell script that uses 'entr' to watch those files and pop up notifications when tests fail.
I still don't know the root cause, but now I know one more place to look for heisenbugs.
In short, you set up a global hotkey to trigger the rerun of a "key"-ed command. Then you can quickly run a command, which can be rerun with that hotkey.
So, a global hotkey set up for `runrerun -r foo`
Then in a terminal you can use `runrerun -b foo -- <COMMAND>`.
It is a bit over-engineered with multiple sessions, different kill signals for restarts, etc. But that's the gist of it.
> broot solves this problem by being aware of the size of your terminal window and adapting its output to fit it.
What? You can just pipe `tree` into `less`. The solution isn't to download Yet Another Binary that does this specific thing and doesn't come installed on most platforms.
Which makes sense, this is an article for people on Apple hardware (judging by the installation instructions), so no interest in keeping things "the unix way of doing things"
I'd argue that Linux is more Unixy than macOS in terms of philosophy though, I remember reading that Dennis Ritchie considered Linux as much of a Unix as the other offerings on the market at the time which were the BSDs and commercial Unix variants; his opinion is obviously worth thousands of mine!
/s
ls **(.)
To list all files, recursively, or some variation thereof depending what I want: *(/) for directories, *() for executable files, etc. With a global alias it can be piped to less with LL: "ls *(.) LL". You can actually do a lot of stuff with it, and is basically good replacement for find, except with less typing. The syntax looks a bit daunting at first, but especially for basic stuff it's not that hard (e.g. / and is the same as what ls -F uses).I wish there was a better pager than less though, but haven't really found it (and no, Rust project X with 115 different colours that don't even work on a light background and/or "fuzzy" matching that never gives me what I want is not it).
Have you ever tried to pipe `tree` into `less` in a real code directory, with directories containing thousands of files ? That solves no problem.
The idea of broot isn't just to "be aware of the size of your terminal": its view is BFS based to show you a useful overview.
See explanation here: https://dystroy.org/broot/#get-an-overview-of-a-directory-ev...
My guess is that author wanted history-search-{for,back}ward in their .inputrc for up-down keys but missed that somehow. The default history behavior is undoubtedly tedious. https://unix.stackexchange.com/a/20830
The only thing I’d do to history-search- is to show a list of variants (like in wildmode=list:longest or similar) instead of presenting them one by one. Pretty sure it is already possible.
Once I found this, I rarely use Ctrl+R since most of the time I'm searching based on starting command name.
If I'm using something a lot of times, it gets added as alias/function. I also maintain a file with commands that I think may be useful later, easier to search that compared to online search.
Yes! I have OBTF with all “devops/etc” stuff that is hard to remember once a quarter, with headers and explanations, and everything in select-paste ready format.
alias h="history | fzy"
alias f="fd | fzy"I may be unique in this regard but the types of actions I do in one path are very different from those I do in others: working on my blog is very different from the types of things I do when working on code and it was nice to have history aware of this.
I haven't actually used McFly a lot though, so I haven't quite seen if it going to become a permanent thing for me. The UI of using FZF for history is nicer, I just didn't like the suggested matches.
You are right though, I don't have my arrow keys setup like this. I'll try that out as well. Thanks for the tip!
My point being: only replace your weapons of choice once you are confident you are using them to their full potential and you identify needs that these known tools wont solve.
If you are a novice, you don't start with alternative tools, because they are much more likely to lose popularity, stop being developed, and lose relevance.
Same with fzf. On a Mac, copying specific output from a command becomes so easy. For example `git branch | fzf | pbcopy`.
Recently I started using vscode with the remote-ssh plugin and it has surprisingly been a pleasant experience. I have an integrated terminal to the remote box and I have full intellisense for coding (I do 80% coding / 20% admin, probably).
It took a gestalt flip for me to realize that with this plugin installed, vscode is basically a very (very) fancy terminal emulator.
Only problems I've encountered so far are if you paste large strings or quickly scroll through your bash history, sometimes characters will drop from the strings. Also, sometimes you have to reload the window because intellisense stops working because it seems like the language server installed on the remote box stops working.
Think before you downvote - is this incorrect, really?
I use multiple spaces in macOS, with entirely different project contexts. It helped having the labeled terminal tabs coupled to a Pycharm project window.
However, this also let me to realize that the gate commands I was using could be accomplished through pycharms interactive source code management tools.
This helped me realize that there were more powerful things I could do there than in a terminal.
I still find myself using terminal to debug remote environments. But for building, the IDE has taken over over for a lot of my past terminal use.
Pipe is the new-cool! (always_was.jpg)
FZF, for instance, is a major novelty in terms of interface, and it really follows the UNIX philosophy and shows immense composability.
Another big upgrade was dmenu. Just slap it in some bash script and now it's interactive. Combined with xbindkeys you can even bind those menus to keys, and it'll work on any Xorg-based desktop.
There are those edge cases where i want to edit and get something out of a unix pipe based oneliner i made however in certain scenarios i find myself having to write complicated code for just a one or two line edits to the piped output
In those i recommend using vipe (its a part of moreutils) it lets you put in your favourite text editor in between your unix pipes , so you can edit things manually from the stdout midway a pipe and let the rest of the pipe take it from there.
Its quite handy , this way i can sometimes use vim’s macro functionality inside my pipe oneliner scripts for when its not worth it to code it with sed,cut,awk or grep.
tree | lessShow me files in this dir:
$ n
Show/hide details for each: .
Filter / search file list: /
Navigate subdirs (or parents) with vim or arrow keys.Cd into sub dirs by typing unique chars from dir names:
ctrl-n
Drop back to shell i started n from but cd into this new dir: ctrl-g
Exit n without cd’ing to the new dir i navigated to: q
https://github.com/craigjperry2/dotfiles/blob/main/dotfiles/...Although for deeply nested targets where i already know what im after, i’ll just:
vim<ctrl+t>
To launch an fzf fuzzy finder on the (.gitignore aware) results of an “fd” (cross platform alternative to gnu and bsd find)It seems like they even have a neovim plugin, you might consider using it.
That said, i don't find myself using that as much. Usually i'm in the shell when i invoke nnn - i might open a file in vim from nnn though.
In vim, i typically lean on fzf.vim more often - usually i know something about the next file i want to open so it just feels more direct.
Edit: That and the speed. And since I am more proficient with Python, hacking on Ranger is easier.
But - if you're happy with ranger, i'm not sure it's worth the switch - they're very similar. nnn is quite a bit faster than ranger but other than that, i think ranger has more community support.
Shameless plug but I'd written my own $SHELL callewd `murex` as I kept running into pain points with Bash as a DevOps engineer. The shell doesn't have `tree` inbuilt but it does have FZF-like navigation built in.
https://github.com/lmorg/murex
I've been using it as my primary shell for a few years now and I'm not going to pretend that it isn't BETA it does work. However it's not POSIX and some of the design decisions might rub people the wrong way (given how opinionated peoples work-flows are). But if you're curious then check it out.
e.g.
ps -A | grep ruby | grep fsevent | ruby-each-line "puts l.split.first" | xargs kill -9
cat c | ruby-each-line "puts l.split.first[0..-2]" >> .env.development
pbpaste | ruby-each-line "puts l.split.last" | ruby-all-lines "puts lines.map(&:strip).join(',')"> ps -A | grep ruby | grep fsevent | ruby-each-line "puts l.split.first" | xargs kill -9
killall -9 -r ruby.\*fsevent
> cat c | ruby-each-line "puts l.split.first[0..-2]" >> .env.development awk '{ print(substr($1, 0, length($1) - 1)) }' c >> .env.development
or awk '{ print $1 }' c | rev | cut -c2- | rev >> .env.development
> pbpaste | ruby-each-line "puts l.split.last" | ruby-all-lines "puts lines.map(&:strip).join(',')" pbpaste | awk '{ print $NF }' | paste -sd,
(I couldn't work out exactly what the last two did so I'm not sure my translation is accurate. Even if it's not quite right I doubt the intent would be impossible to express with basic tools.)After all, there's just one person using your shell: you. So whatever works well for you is a "good" solution.
> I never really learned awk properly myself either
It's really worth it. It's simple — pattern matches and block, BEGIN/END, hashes, match() and gsub() cover 90% of my uses. I very often find myself writing something like this:
awk -v id=$id '
$1 == id {
go = 1
}
{
if (go && $4 == "b667226") {
km += $5
secs += $6
n++
}
}
END {
printf("%d rides, %d km (approx %.1f hours)\n", n, km / 1000, secs / 3600)
}'
This program takes STDIN, reads each line and:* sets a flag indicating interesting data has been found if the first column matches an ID passes on the command line * if the flag is set and column 4 contains "b667226", add columns 5 and 6 to running totals, and add 1 to a count of matching lines * after all the input has been read, prints out a summary of the data
Of course, any language can do something like this, but awk is succinct, easy to iterate on, and available almost everywhere.
This week I've been doing some Lua programming; I had done some Lua several years ago for something else, but found that I had forgotten most things. Even for Ruby I've forgotten quite a lot, yet for two years I programmed Ruby every day for a living. It's just that I haven't done much Ruby since, and when I did some Ruby several months ago I had to look up quite a lot of basic syntax things because I had roughly remembered how it worked but not enough to actually get stuff done in it.
In general, I find that effective practical programming skills are a bit of a "use it or lose it" thing. I don't think I'll use awk enough to not "lose it", even though it's a fairly small language, and most problems it solves can also be solved in other ways.
I highly recommend it, and I consider it by far the most useful in this list!
I use "ESC-c" a lot. It calls fzf and fd, you type a part of your directory name, hit enter, and it cd's to that directory.
export FZF_ALT_C_COMMAND='fd --type directory'
That's in my .zshenv.It's incredibly useful if you have multi-layered project directories. Like ~/Projects/Terraform/Cool_Project - you'd hit esc-c, type cool, hit enter and you're in the place you wanted to be.
Be careful with quoting though, got myself into a situation where every new terminal asked for the root password. Shellcheck found the reason quickly.
(edit) Another benefit is that those aliases abstract over differences of invocation on different distros. Here are some examples:
# PACKAGE MANAGEMENT
alias pcl="sudo zypper cc && sudo zypper purge-kernels"
alias pin="sudo zypper install -y"
alias pla="zypper search"
alias pli="zypper packages --installed-only | rg "
alias plu="zypper list-updates && zypper list-patches"
alias pre="sudo zypper remove"
alias pup="far too long" # upgrade all packages and ask for confirmation
alias pups="far too long + shutdown" # upgrade all packages without confirmation && shut down
# QUICK EDITS
alias als="nvim ~/.alias && source ~/.alias"
alias brc="nvim ~/.bashrc && source ~/.bashrc"
alias egr="sudo nvim /etc/default/grub && sudo update-grub"
alias fst="sudo nvim /etc/fstab"
alias pro="nvim ~/.profile && source ~/.profile"
# QUICK MOVEMENTS
alias ..="cd .."
alias ...="cd ../.."
alias ....="cd ../../.."
alias sc="shellcheck"
# SYSTEMD
alias sden="sudo systemctl enable --now"
alias sdls="sudo systemctl --type=service"
alias sds="sudo systemctl start"
# VIM
alias nrc="cd ~/.config/nvim/ && nvim init.lua"
alias nv="nvim"
alias scratch="nvim scratch.txt"
function ..() {
for i in $(seq 1 $1); do cd ..; done
}
which enables .. # up 1 dir
.. 2 # up 2 dirs
.. 42 # up 42 dirs
Here are some more: https://pilabor.com/blog/2021/03/unix-shell-tricks/ function ..() {
cd $(printf '../%.0s' $(seq 1 $1))
} # Expand ... to ../..
function vbe-expand-dot-to-parent-directory-path() {
case $LBUFFER in
(./..|* ./..) LBUFFER+='.' ;; # In Go: "go list ./..."
(..|*[ /=]..) LBUFFER+='/..' ;;
(*) LBUFFER+='.' ;;
esac
}
zle -N vbe-expand-dot-to-parent-directory-path
bindkey "." vbe-expand-dot-to-parent-directory-path
bindkey -M isearch "." self-insertIf you are on
/home/user/projects/app/src/handlers/
and you want to go to `app/`, you'll do: > up src
Or give name of any other child folder/file of `app/`. So if `.git` exists in `app/`, > up .git
would work too.I made it for Fish shell: https://gist.github.com/ajitid/81a4993be410586c038f8b3fc140b...
> Note that the shell will attempt to change directory without requiring cd if the name of a directory is provided (starting with ., / or ~, or ending with /).
> https://fishshell.com/docs/current/cmds/cd.html
So these all work without the cd by default with Fish:
../
../../
~
~/path/to/folder
cdh is also very useful. It shows a history of recent directories with a keyed prompt to jump to a directory.
-sox, audio swiss knife
-ffmpeg, ditto for video
-ncdu, check disk usage
-rclone, mount and rsync everything
-f3, check and forbid overwriting those fake USB drives
-udfclient UDF, but better as it eases compabitibility. Ditch fat32/NTFS for computer media sharing
-trickle For network/web programmers, it can force really slow connections on software, such as 2G/ISDN speeds and even below. Perfect to test bad conditions
-pdftotext -layout It dumps a whole PDF into a text file. Useful for copy/paste or adapting the format to anything else - unzip -qcp "$EBOOK_BOOK" "ml" ".htm" | lynx -force-html -dump -stdin -nolist > book.txt EPUB to utf8 dumper.
Trivially adaptable to be used with less, for example. Use "$1" instead of "$EPUB_BOOK" to use it on a script.
git config --global alias.wip "\!git add --all; git commit -a -m \"wip: update\""
git config --global alias.wtc "\!git add --all; git commit -a -m \"[WTC] \$(curl -s whatthecommit.com/index.txt)\""
I don't like the idea to generate the list of modified files in commit messages, it's not very readable to me plus that I could just generate such info with git log. I tried to write a custom git-wip script to include output of "git status --porcelain=v1" but turns out it's just not necessary since "git log --name-status" could already show it git log --name-status
# or get modified history of a specified file
git log -p path/to/file.extI recently switched to the latest neovim and redid my config from scratch using more modern plugins. Took a while to understand the ecosystem and figure out how to get things working together nicely. But now, between telescope, ripgrep, trouble and better overall LSP etc, I’ve gone from fast to faster.
The balance of payoff probably still isn’t there yet, but it’s made me want to code more.
Fish shell has this built in. The search history (by pressing up) depends on your current directory. It's very useful.
Basically to change anything about any request sent or response received.
export FZF_ALT_C_COMMAND='fd --type directory'
That's in my .zshenv.It's incredibly useful if you have multi-layered project directories. Like ~/Projects/Terraform/Cool_Project - you'd hit esc-c, type cool, hit enter and you're in the place you wanted to be.
- broot - like tree
- funky - like direnv
- fzf - fuzzy finder
- mcfly - cli completion
- zoxide - like cd
- gitupdate - auto commit and push
The author references this discussion https://lobste.rs/s/yfgwjr/what_interesting_command_line_too...
jq is also super nice when you have to parse json data.
I also like powerlevel10k: https://github.com/romkatv/powerlevel10k
<run some command that outputs json> | jq .
and you'll get nice color pretty printed output. 90% of the time I use jq it's just to pretty print like this.
installed everywhere
You have to configure it yourself, but it works on bash, fish and zsh.
If I recall correctly, you would open up the repl and start typing a command, and the output would refresh as you typed. Does this ring a bell for anyone?
If you are using fixup commits in your PR review process it's essential.
How do you type name into the terminal?
Forking with gh is pretty simple:
Clone the repo, cd to it, "gh repo fork"