Shell Productivity Tips and Tricks | Hacker News Reader

Shell Productivity Tips and Tricks

blog.balthazar-rouberol.com

353 pointsgoranmoomin96 comments

Today, was actually the first time that I used the vim time travel feature. I noticed that I had deleted a line by accident somewhen during the last few minutes. So I typed

  :earlier 5m

and copied the line. Afterward, I typed

  :later 5m

and was back where I started, ready to paste the missing line :-)

On mornings when I'm feeling particularly lazy in the office, I'll do

  :later 8h

and commit the code before heading home. ;)

The single best shell productivity tip for me was to use z:

https://github.com/rupa/z

It automatically makes a list of directories you frequently use in your shell and if you type `z a-small-substring-of-the-directory-path` it does a cd to that directory. (e.g. I type `z and` and it changes my current directory to $HOME/Work/Projects/android`). Saved me tons of typing.

A somewhat less fancy version of this is $CDPATH.

   export CDPATH="/path/to/my/projects:$HOME/Work/Projects"
   # From anywhere.
   cd andr<tab><enter>

Note that CDPATH can break shell scripts that assume `cd` hasn't been adulterated.

A huge block of "unalias" calls at the beginning of shell scripts used to be a common practice, but they don't seem to be found in more widely-used tools these days.

Except with CDPATH the `cd` is usually not an alias. Even `/usr/bin/cd` is aware of CDPATH.

It's a bug in the script, just like reacting badly to $IFS or $LANG.

What is /usr/bin/cd for? cd is supposed to be a shell built-in, because running /usr/bin/cd will make another process and that cannot change the current working directory of the shell you are in.

EDIT:

  $ cat /usr/bin/cd
  #!/usr/bin/sh
  builtin cd "$@"

Look, it's 'cperciva!

  $ cat /usr/bin/cd
  #!/bin/sh
  # $FreeBSD: src/usr.bin/alias/generic.sh,v 1.2 2005/10/24 22:32:19 cperciva Exp $
  # This file is in the public domain.
  builtin `echo ${0##*/} | tr \[:upper:] \[:lower:]` ${1+"$@"}

TIL, magnificient! It doesn't work on the minimal Centos 7, so you need:

    yum install -y bash-completion
    bash      # a fresh run

CDPATH doesn't require the export as it is local to the shell session. Simply setting it will work as you'd want and won't pollute the environment for subshells and scripts.

And zsh users get another advantage here. You can use assign to the cdpath array or the CDPATH colon-delimited string, they are both treated the same internally. The advantage is being able to `cdpath+=...`, much like adding to $PATH with `path+=` or removing from PATH with `path^=`. Check zshbuiltins(1) for how to make your variables work like this, or how to use typeset -U to remove duplicates automatically, or $thousand_other_cool_things.

Agree, so useful! A port is also available for PowerShell: https://github.com/vincpa/z

Have you checked out Warp drive for zsh? https://github.com/mfaerevaag/wd It is more manual than z, but takes out the learning phase. Will check out z though!

It depends on how you work, I have a lot of small projects which come and go, so setting up the paths every time I start or finish something would be inefficient.

Although I might adopt it for different weird system configuration directories which I often forget the path to.

Yes I have had that problem already, more specifically cleaning the lists in wd if directories are moved or removed. I do a cronjob to try to rectify that but interested to see if z mitigates this problem - seems there is an option -x to remove learned directories.

[edit] Or z's based ageing ranking system will just aged those legacy directories, even better (in terms of more hands off)!

Sound similar to autojump. https://github.com/wting/autojump

The advantage of autojump - at least for me - is that it doesn't depend on the shell, as it is implemented in Python. This makes it possible to read and write the data elsewhere, too, independent of your shell.

For example, I integrated it into ranger (see my history, I don't want to spam HN with this project).

I use autojump, but I liked the idea of having an fzf-powered menu pop up when invoked without any args (which is not how autojump behaves). So I made rustled this up (zsh): https://pastebin.com/aLmqJVRy

This is my workhorse, 'to'.

No argument: start fzf (this really should read "to $(fzf)"), jump to that dir.

One arg: if the path is a real file/dir, jump to it, else use zoxide (z in rust) to soft-match it from history.

https://github.com/xkortex/bashbox/blob/master/aliases/core....

I rarely use Ctrl+r to get command from history, because most of the time I know the starting characters of the command I need. So, I type those characters and use arrows keys to match from history. This is enabled via these settings in .inputrc

    # use up and down arrow to match search history based on typed starting text
    "\e[A": history-search-backward
    "\e[B": history-search-forward

I also changed couple of setting wrt Tab autocompletion:

    # when using Tab for completion, ignore case
    set completion-ignore-case on
    # single Tab press will complete if unique, display multiple completions otherwise
    set show-all-if-ambiguous on

You can use ctrl+r for commands which are similar (say ssh to n number of servers, so the up/down would still take some amount of browsing the options)

Just tag commands with a # at the end

> $ complex command here # tagHere

Later use the tag with ctrl+r

I do this all the time

> ssh 127.0.0.1 # master

yep, ctrl+r is better suited for such cases or whenever you have to match a command other than starting characters

and I think there are some fuzzy matching options (like fzf) that can be added to ctrl+r

When I used bash, I had all of your settings on (had more, some others I can live without - these four I can't). I still have them in .inputrc for programs that use readline!

I have since switched to zsh. Up/down to search history, in ~/.zshrc:

    [[ -n "${key[Up]}" ]] && bindkey "${key[Up]}" history-beginning-search-backward                                              
    [[ -n "${key[Down]}" ]] && bindkey "${key[Down]}" history-beginning-search-forward

Autocompletion in zsh is a different beast altogether :)

fishshell is my favorite productivity enhancer for the terminal: https://fishshell.com/

The primary reason I chose over zsh/bash was that autocompletion just works. If you install it, I highly recommend oh-my-fish as well, which allows you to add packages for look/functionality: https://github.com/oh-my-fish/oh-my-fish

If you're a zsh user, they have a similar framework: oh-my-zsh: https://github.com/ohmyzsh/ohmyzsh

I really enjoyed using fish for a short while, but soon I learned that it was not POSIX compliant. This was a deal breaker for me since my colleagues would often share scripts that wouldn't just work for me. I use zsh nowadays.

I love zsh, but it isn't without its own strange quirks that make compatibility exciting at times. In a thread from a fortnight ago¹, RANDOM is handled very differently for example. Try `echo $(echo $RANDOM) $(echo $RANDOM)` in zsh and bash.

I'm unsure whether minor differences that catch you out occasionally are worse than huge differences that you always have to think about. For me zsh still wins ;)

1. https://news.ycombinator.com/item?id=22841259

Wow! Even repeating that call always yields the same result in zsh - as long as it's a subshell.

I think your comment will save my sorry ass one day.

Well, you could always just use a shebang or call bash/sh directly on the file.

Without the same fluency in bashism from using it for day-to-day activities, the colleague provided script is less readable and less editable for the fish-fluent, bash-inarticulate. Any fish-specific scripts are also of low-value to colleagues.

It's not that it's impossible to be a shell polyglot, but GP deemed it easier to learn !fish.

I tried fish for a while and loved the history recommendation mechanism, but the 'vi mode' emulation had enough annoyances that I went back to zsh. From the bugs filed on it, it seems like the vi fixes weren't really possible.

I did install a similar history recommender in zsh and have been pretty happy with that.

Another vote for fish. I would say another benefit is that it has a reasonably great out of the box experience with little customization needed to get a lot of its benefits, eg awesome autocomplete.

The downsides have been mentioned, namely the lack of compatibility with shell scripts not written for it. In the same vein, until somewhat recently, operators like && did not work for it.

It’s still my daily shell, and I really only use something like bash when having to remote into a production server.

My productivity stack:

Zsh (with all the autocomplete plugins)

Oh-my-zsh zsh-autosuggestions: https://github.com/zsh-users/zsh-autosuggestions

Tmux

Histdb: https://github.com/larkery/zsh-histdb

Zoxide (replace cd): https://github.com/ajeetdsouza/zoxide

Fzf

Rip (replace rm): https://github.com/nivekuil/rip

My personal dotfiles: https://github.com/xkortex/bashbox

Mosh (replaces/augments ssh)

What I'm currently looking for is a better multi-host logging solution.

> If you want to remove a sensitive command from your history, you can simply edit your $HISTFILE history file and remove it.

It’s already too late at this point. Anybody on your machine can read the commands you are running, for example with ps. You should instinctively avoid entering anything in plaintext into a terminal which you don’t want other people to see. Any command line tool worth its salt will provide alternative ways of passing secrets (using configuration files for example).

How exactly?

I think what OP meant was closer to "if it could have been entered into the bash history than there's other ways it could be seen".

Most CLI programs that need sensetive information as input should either do it interactively (a la sudo), as standard input, or configuration file. If worst comes to very worst, you can put the sensitive info in a text file and use backticks. For example:

  some-command --username=jedimastert --password=`cat secret.txt`

How would backticks help with hiding it from ps?

I thought that's how it would show up in ps. Apparently I was incorrect. :(

> some-command --username=jedimastert --password=`cat secret.txt`

This will not work in the way that you suggest. The output of ps will show the result of `cat secret.txt` and thus reveal the password.

Would it? Poo. That's my bad.

Not a real solution, but you can assign new values to argv/argc to hide things from ps.

With Perl, for example, assign to $0.

$0="what /bin/ps will see";

The worst is when you expect a password prompt so you type the password and hit enter, but you mistyped the original command and you end up typing your password in plaintext onto the command line. oops.

If you're using `bash`, A quick remedy for this is:

  unset HISTFILE
  exit

This prevents `bash` from updating the `~/.bash_history` file on exit with the command you don't want memorialized. (assuming you don't also have some hook that updates your history file after every command)

I understand what you're saying but if anybody on your machine can read the commands, can't they also read your configuration files? Or are you saying a different user on the machine can run `ps` and see my processes but not necessarily my portion of the file system? My thought is if the file is not encrypted on the disk, then any desktop application can read it. So, while I agree that preventing a user from reading bash history is not worth it if they can read your processes, aren't configuration files (if unencrypted) just as insecure?

In a Unix system, files have 3 levels of permissions — for the user who owns the file, the group-of-users that owns the file, and any system user.

So, a given config file can have permissions so that the file owner can read and write, but other users cannot. Like your ssh keys.

But ‘ps’ can be run by anyone, and it can typically access the whole command line you used.

This is true but in a typical desktop usecase there is one user.

Ah, right. This makes a lot of sense. Thank you!

> Anybody on your machine can read the commands you are running, for example with ps.

This is why I always mount /proc with the "hidepid=2" option (as well as the "gid=<n>" option to ensure that doesn't apply to the members of a specific group, such as the "admins" of the box).

This doesn't have to be true. You can mount /proc with hidepid to prevent users from seeing other users' processes: https://access.redhat.com/documentation/en-us/red_hat_enterp...

Indeed! One thing that at least works on zsh is to prefix the command with a space, then the command doesn't go into the history file.

So if you do `date`, it'll end up in the history file but if you do ` date` (one space prefixed), it won't. One `clear` later and it's like the command was never run.

Two big ones missing. After ctl-r history search:

  ctl-o ctl-o ctl-o   re-execute the history one by one
  
  alt-. alt-. alt-.   last argument (better ux than that !$ business)

I didn't know either of those, thanks!

Some plugins that saved me lots of typing with zsh:

https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/histo... looks through your history and only shows entries that started with the same characters that you already typed. Similar to Ctrl-R with fzf, but still similar enough that I use both.

https://github.com/zsh-users/zsh-autosuggestions also looks through your history for commands with the same start, but it will automatically suggest them for you to use. If you often use the same commands, this saves quite some typing. For bonus points, I hooked it up so that Ctrl-Return will execute the suggestion.

https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/per-d... To make the first two even more powerful, I split my history based on the directory I'm in. This means that it's only suggesting me commands I entered before in this directory.

I'm a huge fan of the first 2 plugins, but I didn't know about the per-directory history plugin. I might put that in my toolbox. Thanks for the recommendation!

Process substitution deserves a mention: http://www.tldp.org/LDP/abs/html/process-sub.html

**

is not "expanded to all files and directories in the children directories, with a depth limit of 1." It's hard to see using `ls` because `ls` doesn't just list its arguments, it also lists files inside its directory arguments. Use

  echo **

and you'll see it just expands to the same as

- all the files in the current directory. Or type

**

and press Ctrl-x, Ctrl-* (glob-expand-word) to expand it.

After `shopt -s globstar`

**

expands to all files recursively.

Also, `echo $dir | tr '[:lower:]' '[:upper:]'` should be `printf '%s' "$dir" | tr '[:lower:]' '[:upper:]'` to handle spaces and other weirdness in file names.

(If anyone knows how to escape asterisks in HN it would be helpful to know :)

fzf fizzy history search with ctrl-r is my favorite new shell trick. Fzf is a wonderful program

fzf's great. I use it for a quick/easy git branch menu, complete with preview of recent history of each branch: https://pastebin.com/R91bP2qa

I too prefer fizzy history. It's like Sodastream for your shell!

Skim [0] is a nice alternative to fzf [1], mainly for the fact you can call additional commands dynamically like so [2]:

    sk --ansi -i -c 'rg --color=always --line-number "{}"'

[0] https://github.com/lotabout/skim [1] https://github.com/lotabout/skim#difference-to-fzf [2] https://github.com/lotabout/skim#interactive-mode

Would that correspond to piping the output of the additional command to fzf?

I like fd more. fish already has autocomplete.

No matter how many times I read articles like this I always manage to learn something new. For example this time I learned about brace expansion. Will save me retyping a bunch of commands with different text.

Same here. I though I'd seen most of them by now but apparently not :)

echo ${MY_VAR:-some_default} is another class of useful functionality.

sadly, i know how that works only because it was the recommended long term fix from an RCA

If you care about your command history, I would advise against doing this unless you are truly a sed expert:

    $ sed -i '/secret-command/d' $HISTFILE  # deletion of history line containing 'secret-command'

Because if you get it even slightly wrong, you could easily hose your entire history. If it were me, I would just use an editor to delete that line (or two, if timestamps are enabled).

And if you know don't want a command in your history, prefix it with a space.

xargs should be there. The most useful tool IMO.

https://en.wikipedia.org/wiki/Xargs

As zsh is mentioned at various points throughout the article I'll add that zargs¹ is often a great alternative to xargs. Combined with zsh's awesome globbing and filtering functionality it is often far nicer to work with than find.

1. https://github.com/zsh-users/zsh/blob/master/Functions/Misc/... or zshcontrib(1)

I think zsh is the best shell ever, unfortunately, I've reverted to bash, because it's too much hassle to install zsh everywhere I work. zsh globbing is on another level.

> zsh globbing is on another level.

I agree. That said I'll freely admit that liberal use of glob modifiers and qualifiers can lead to the appearance of line noise. Because of that I'll occasionally unroll them with `find(1)`-syntax if I post a snippet to help a co-worker, but love their expressiveness when in an interactive session.

For people not familiar with the extensive globbing options of zsh see https://linux.die.net/man/1/zshexpn .

you may want to look at https://github.com/romkatv/zsh-bin. roman also has some tooling which helps when ssh-ing to remote servers without zsh.

On more handy set of emacs-derived keystrokes:

Ctrl-a ctrl-k to clear the current command, or redo a hidden password prompt if you know you fat-fingered it.

ctrl-a jumps the cursor to the beginning of the line, and ctrl-k deletes from current position to the end of the line.

The mnemonic to remember these: "a" is the first position in the alphabet, and jumps you to the first position. "K" is short for "kill.”

Ctrl-u (delete everything from the cursor to the beginning of line) is one less stroke for the very same result :)

Ctrl+U is also really handy for clearing password (or any other input) lines where you know you made a mistake, but don't want to just spam Backspace until you think you've gotten to the beginning.

Any ideas why ctrl+k doesn't work in my iTerm2? All other readline shortcuts work.

The article links to this at the bottom, but just calling it out again because I think it's so useful. I am using git all the time---probably a majority of my terminal relates to git in some way. I think the git bash completion script is extremely useful. https://github.com/git/git/tree/master/contrib/completion

It's incredibly easy (and transparent) to "install" and saves a ton of keystrokes.

Best shell productivity trick ever: don't use the shell for anything more complicated than launching executables and use a reasonably modern programming language to achieve all other tasks.

What is a reasonably modern language? Everything seems to be at least 20 years old now.

OP might have made their statements tongue in cheek.

But there's the truth in there that even programming languages that are 20 years old had a completely different mindset applied to their design process than the original Unix shell which was much more ad-hoc.

40 years ago IT was a completey different environment than 20 years ago.

Not tongue in cheek at all, python might be 20 years old but is continually developed, has come all the way to have a typing system and has a huge ecosystem for all things devops

no thanks.

'rename .txt _1.txt *' is quite a few lines in a language that's super simple, like Python, and it turns into code golf with even more terse languages.

I'll stick to the terminal.

> anything more complicated than launching executables

mv is a simple executable. GP is talking about stuff like generating reports or sophisticated orchestration in .sh

I'd love a good answer here. Shell is clunky and annoying as hell, but a "better" language like python is very verbose for typical shell stuff like process management (processes, pipes, signal handling), small text handling tasks like regex matching or string splitting.

Perl is maybe the closest here, but it certainly has it's fair share of annoyances, and I more or less dropped perl for python decades ago and don't particularly feel like going back.

Not really, with python there's a habbit-acquiring curve but you end up with longer lasting more readable and maintainable assets that can be modularized refactored etc. perl is write-only and has little going for it afaic but ymmv.

set -o vi to use vi on the command line.

Whoever can point me at a robust method to save/restore bash histories of a tmux session would save my life!!! [And yes i use tmux inside tmux]

I use this to have all shells share a history concurrently - which sometimes is confusing if you press "↑" and have an unexpected (possibly dangerous command) but I'm used to it now:

  shopt -s histappend
  PROMPT_COMMAND="history -a; history -c; history -r"

See more about it here: https://unix.stackexchange.com/questions/1288/preserve-bash-...

About a year ago I went further into this rabbit hole and right now I think I am on a re-iteration of a system I have not changed in about 4 months. It came to me after watching too much sci-fi that had clicky-clicky-typie-typie instant recall.

It occurred to me that every hack around history just nibbles at the edges probably because all of them are bolted on a design when the systems were slow. Systems now are fast and have good connectivity ( at least to themselves ) hence:

"All logins across all systems that have the same security domain share single common command history."

I use bash but it should be adoptable to anything. Here's how it works:

1. There's a database. I use redis because it is fast and has SETNX. [I may switch this to a redis queue to be able to archive commands as well.]

2. Using PROMPT_COMMAND variable I execute a function that pulls the last executed command using "history 1" and pushes it into the database. SETNX ensures that only a command that has not been seen before is stored. In the same transaction I trim the number of commands stored in the database to 5000.

3. Ctrl-R is bound to fetching last 5000 commands from the database, piping it into "peco" which I use with the selection line on a top ("fzf" does not support the top line so I don't use it here. "peco" does not support certain header skips/treatments, so i use "fzf" in other places )

4. On a failure to push into the database, system does nothing as the data is in a local history.

5. On a failure to pull from the database, system feeds the result of the local history into the "peco".

This system gives me ability to access the same command line history across my laptops, workstation, VMs, random Linux devices that i have running at home. Our ops-team uses it across all the monitoring workstations

I'm using a variant of this: https://spin.atomicobject.com/2016/05/28/log-bash-history/

Logging the entire bash history in files sorted by day: export PROMPT_COMMAND='if [ "$(id -u)" -ne 0 ]; then echo "$(date "+%Y-%m-%d.%H:%M:%S") $(pwd) $(history 1)" >> ~/.logs/bash-history-$(date "+%Y-%m-%d").log; fi'

I haven't overloaded ctrl-r yet to search in these files but that would be a possibility.

Histdb is amazing, but requires zsh. I have one continuous history. I'm working on figuring out multi-host history for One True History.

https://github.com/larkery/zsh-histdb

I haven't tried it, but the keywords you are looking for is bash history db/sqlite. E.g.:

https://github.com/thenewwazoo/bash-history-sqlite/blob/mast...

> Toggle your cursor between its current position and the beginning of line

That's quite not what ^X^X does. The bash man page says:

Swap the point with the mark. The current cursor position is set to the saved position, and the old cursor position is saved as the mark.

I max out my history sizes as well as have bash functions to dump/persist them to time stamped archives on command or on logout.

(That’s

  export HISTSIZE=
  export HISTFILESIZE=

for those wondering how to do it.)

My productivity tip:

Sit down, and read through `man bash`.

Thank me later.

Not built-in but hstr (https://github.com/dvorka/hstr) and tldr (https://github.com/tldr-pages/tldr) are awesome.

vi :x

grep -n ... vi +§

Reply on news.ycombinator.com