Better Bash History (2012) | Hacker News Reader

Better Bash History (2012)

sanctum.geek.nz

267 pointsPete_D83 comments

I set up my bash config to store history separately for each working directory (ie separate history files that I store under a $HOME/.bash_history_d directory) – so as I cd into a project directory, the history is populated with the commands that I've run within that directory.

I find this super-helpful to remind me of the context for projects – cd in, then up-arrow to whatever that command is I usually run to launch this server, grep this logfile, etc.

There are some downsides – sometimes I want to run the same command in directory X and also in subdirectory X/Y, but I love doing it this way.

Anyone else done something similar?

That's clever, I'd like that. Can you have another key to recall the global history, e.g. all the combined histories like you would normally see?

I do something similar, but instead of current directory I have a notion of context that is carried by an environment variable set in a GNU screen process (and therefore inherited by its children) which drives logic in my .bashrc.

https://github.com/dlthomas/config-files

Fish does this by default (and much more as well).

Could you elaborate? I can read docs, but it's not clear to me what correspondences you have in mind.

Specifically, I'm not sure what "this" is that fish does, as I described interaction of multiple pieces of software and I don't think fish includes a terminal multiplexer?

autocompletion in fish takes into account the directory where you are at, so it will autocomplete commands ran in the same directory before other commands.

If that's what's being referred to, it sounds like dual_base meant to reply to aaronharnly? As mentioned, my setup isn't based on working directory.

Oh, I see. I interpreted "context" as current working directory. I took a look at your .bashrc but I couldn't detemine any sort of partitioned history. Are you saying that you have something like a virtual environment (as in Python) and the bash command history is separated according to the environment?

I split up my bashrc, pulling in different "component" files: https://github.com/dlthomas/config-files/blob/master/.bashrc...

The history file is set here: https://github.com/dlthomas/config-files/blob/master/.bash.d...

I use zsh, and I have a thing in my profile that skips writing any `cd` command to history. Instead, it will rewrite the command using the absolute path of the directory I `cd`'d into and write that to history.

Thus, all of the `cd` commands that get written to my history use absolute paths and are re-usable no matter where I am.

It would be cool if this sort of functionality were generalizable to all commands where you type a relative path, not just `cd`.

https://github.com/dasl-/settings/blob/3143bbfe23bd75c3e35c7...

You might enjoy checking out 'fasd', which, among other things, creates an alias ('zz') with autocomplete functionality for changing to frequently-used directories.

https://github.com/clvv/fasd

Or just github.com/rupa/z

In fish, suggested autocompletions are contextual. For example, if I am in /home/dual_basis/projectA and I type `cd some/subdir` then a bunch of time goes by when I come back to /home/dual_basis/projectA and start typing `cd` it will suggest `cd some/subdir`, even if I have typed a bunch of other `cd` commands while in other directories.

This isn't specific to `cd` commands either. It's useful if you are `rsync`ing different directories to different places, or checkout different `git` branches, etc.

That seems like it would be tricky for it not to screw something up.

Like, for example, would it handle this properly?

$ (cd /source;tar -cf - .) | (cd /dest;tar -xf -)

I've done the same thing, and I've found it super helpful.

My particular configuration is at https://github.com/YenTheFirst/dotfiles/blob/master/.bash_cd...

90% of the time, the bash history is super relevant to the current context. A small amount of the time, I remember that I issued a command, but not in which directory - in these cases, I can just search the history directory, and get both the context of what the command was, and also where I was when I originally issued it.

Thanks, reading over that code was useful. In particular I realized that my Oil project (a bash-compatible shell [1]) doesn't support PROMPT_COMMAND, which tons of people appear to be using.

https://github.com/oilshell/oil/issues/320

It seems like a weird hack but apparently it works for tons of things!

[1] Latest status: Success With the Interactive Shell http://www.oilshell.org/blog/2019/02/05.html . Oil runs a bunch of interactive programs like bash-completion, virtualenv, and git-prompt.

This is lovely, and I should integrate it with the chronology based approach I've been using.

I keep an infinite log by appending a timestamp to the beginning of the lines written to my history file, then merging this into a growing log with an hourly cronjob that runs sort -n | uniq on the file and the new version of the shell history. This way the shell history file is just a buffer. It doesn't get too big to cause corruption or other issues with reverse search on the command line. I just grep my way through the big history file to figure out what I was doing and when.

But doesn’t that mess up the order of reverse-i-search?

No because every line has a timestamp and the sort is done comparing numerical value, but still I question the use of uniq, unless it's invoked with the --skip-fields option.

zsh-histdb stores history inside a sqlite database instead of a simple text file.

The good thing is that it includes context: current directory, session, date.

I use it together with ssh-suggestions which can use SQL queries to find the most relevant command. You can program rules such as: use the most frequent command in that session, if you none do it for the current directory in the last 6 months, if not, search globally.

isn't that totally opposite of the *nix philosophy of everything is a text file?

There are plenty of non-text files around. Utmp comes to mind. And the text history is still there.

Thank you for this. It always amazes me when I come across something that would be so useful to me, and yet so obvious in hindsight.

> Anyone else done something similar?

I'm not, but now that you mention this, I think I will. I already use direnv all over the place, and this would be a pretty natural extension. Thanks!

Could you share details on how did you setup this. This sounds super useful

I set PROMPT_CMD to the following function:

    function log_history {
        echo "`date '+%Y-%m-%d %H:%M:%S'` $HOSTNAME:$$ $PWD ($?) `history 1`" >> $MYHISTFILE
    }

which records a timestamp, host name, shell pid, working directory and exit status for each command.

I use one bash history (and fasd history) per tmux session: [link redacted]

It's a great idea. If you additionally also like working with chroots you might find containers an interesting topic.

Basically a container let's you wrap a part of your filesystem, network, process space in a separate world of its own. And in that world you have only the context related stuff like history, processes, libs etc. Additionally most container environments come with tooling that enables you to make everything reproducable, so that you can share your container with others.

The great thing is you can always live in global space, while switching between what that global space means.

How would one go about setting up a container?

`docker run <somethingyoulike>` for instance

To echo what everyone else is saying, get FZF. Most of these linux productivity things are just some dumb pet project that solves a specific problem that a specific person finds annoying. FZF is not this - it makes something you do hundreds of times a day (history and file opening on zsh, bash, fish, whatever) indisputably better. If you find yourself sitting there tapping ctrl+r multiple times like an idiot for some thing that may or may not be there, do your stubborn beardy ass a favour and install fzf.

You can implement some or all of the tips in this HN post, it'll make fzf results better.

Yup, I tried the context aware history for couple of days and it got annoying pretty fast. I have everything in one file which I can just FZF reverse search and bam. No need for infinite up arrows.

Yeah, these kinds of hacks usually end up with unmaintainable mess when cumulated. Just use fzf.

https://github.com/junegunn/fzf for the uninitiated

I love this blog and learned so much about the Unix toolkit and the philosophy behind it by reading it. Too bad it doesn’t seem to be much active anymore, but there’s a lot of great material in the archives that pretty much never gets old since its focused mostly on standard Unix software. The author (Tom Ryder) is an very talented writer with a clear and concise style.

A couple of plugs for your consideration:

- I integrated all of these history options in my project Sensible Bash [1], an attempt at saner Bash defaults

- I made an ebook [2] out of a series which appeared on this blog, titled “Unix as IDE”.

[1]: https://github.com/mrzool/bash-sensible

[2]: https://github.com/mrzool/unix-as-ide

I do:

    promptFunc() {
      # right before prompting for the
      # next command, save the previous
      # command in a file.
      echo "$(date +%Y-%m-%d--%H-%M-%S) \
        $(hostname) $PWD $(history 1)" \
        >> ~/.full_history
    }
    PROMPT_COMMAND=promptFunc

This is in addition to standard bash history, and is much more reliable. Being able to see not only what command I ran but what directory I was in and when I ran it is very useful.

(I wrote about this here: https://www.jefftk.com/p/you-should-be-logging-shell-history)

I do something very similar (just saved in files by month) and it has saved my bacon a few times. I was recently asked to re-run a process that I ran years ago. It would have taken me hours to re-discover how to run the process but instead took a few minutes after grepping the commands.

Every developer should do this.

Yeah, this is how I do it - although I append the hostname to the end of the filename and sync the collection between my machines so that I can grep for history on any machine.

Edit: went to check and apparently I've overcomplicated this somewhat (it's about ten years old) but it is nice if empty lines are not logged. It's an old habit to ctrl-c then hit return repeatedly if a terminal is not responding.

  PROMPT_COMMAND=storehist

  storehist ()
  {
    CMDCNT=`history 1 | sed -e 's/\w* *\(.*\)/\1/'`;
    if [ "$CMDCNT" != "$LASTCMDIND" ] && [ -n "$LASTCMDIND" ]; then
        DATE=`date '+%Y%b%d %H%M'`;
        if [ "$LASTCMDIND" != "$CMDCNT" ]; then
            echo "$DATE $HOSTNAME:$BASHTTY $CMDCNT" >> ~/.custom_history;
        fi;
        LASTCMDIND="$CMDCNT";
    fi;
    LASTCMDIND="$CMDCNT"
  }

Bash's broken history is what finally got me to switch to Zsh. The history behavior you want is: 1. each line of history is appended incrementally, but 2. each shell session only has access to its own history unless you specifically reload the history into your session.

Every combination of history commands in bash didn't do the right thing[1], but Zsh's works as desired.

[1] eg. putting 'history -a' in PROMPT_COMMAND as the article recommends appends your session's entire history each time, IIRC.

See, what I want from my history is every command I have ever run, so that ctrl r can find anything I have done.

Yes what I wrote doesn't exclude that. The only distinction is that you generally don't want all your sessions' histories interleaved. I.e. when you up-arrow, you only want your current session's history, without random things from other active sessions in there too. If you do want "everything you've ever done" including things from other sessions that started after your current session, it's easy to reload your history into your session. I have a 'rlh' (reload history) alias for that.

You always only have your current session history in each bash (unless you reload the history manually) so there is no interleaving.

Yeah I know, I was just making a distinction from cortesoft's "I want everything I've ever done" in my history. I was saying "so do I, except I want active sessions separate" (which as you point out, they are by default).

> [1] eg. putting history -a in PROMPT_COMMAND as the article recommends appends your session's entire history each time, IIRC.

No, only the last command. Very handy because that way you can start a new terminal and have your current history available immediately. I also have an alias r='history -n'. With this you can load the history of another terminal session directly into the current session.

Just tested it and 'history -a' behaves how you say. Maybe it was an older version of bash that had this behavior.

Edit: now I'm obsessing about reproducing this behavior from years ago! If anyone knows what I'm talking about I'd be grateful for some validation :)

`setopt inc_append_history_time` is super useful for Zsh users. This gives you some sharing of histories between shells.

I have a global bash history, having set up bash/zsh to [immediately share without any annoying race conditions.][2] I use FZF mapped to CTRL+R to search through my history.

The unique thing is that I use a [python script (in my dotfiles)][1] to clean the history in a more advanced way than is possible with HISTIGNORE.

The script removes duplicates, leaving the most recent copy and removes lines matching several regexes.

[1]: https://github.com/naggie/dotfiles/blob/master/scripts/clean... [2]: https://github.com/naggie/dotfiles/blob/master/home/.bashrc#...

I was recently planning to write something just like this. Thanks for sharing.

If one is only interested to remove duplicates (your script does more) I've found this one liner in a StackExchange post: awk '!x[$0]++' ~/.zsh_history

Note that this only keeps the first copy, not the most recent copy. To make it so that it only keeps the most recent, you would need to run this through `tac` twice.

highly recommended for fellow ctrl-r addicts: https://github.com/junegunn/fzf

These ones are pretty good too:

https://github.com/cantino/mcfly

https://github.com/dvorka/hstr

I also recommend FZF. Really very handy for doing CTRL-R searches.

I also use it for doing interactive git rebases. I have an alias for doing a fuzzy rebase interactive (frbi):

   frbi = !git rebase -i $(git log --pretty=oneline --color=always | fzf --ansi | cut -d ' ' -f1)^

This is a great idea, thanks for it! I've done similar things with using fzf to git add, git diff, and so on, but I hadn't thought of using it to pick a commit to rebase from.

I made a few changes so I figured I'd post them back here in case you're interested:

    pick-commit = # equivalent to your git log... cut...
    rif = "!f() { rev=$(git pick-commit); [[ $rev ]] && git ri $rev~; }; f"

"rif" = "rebase interactive fzf/fuzzyfind". The function and test just makes it exit cleanly if you escape out of the filter instead of giving "fatal: invalid upstream '~'" (and I changed it from ^ to ~ because I think ~ is usually what one should use over ^, but I'm willing to be corrected). And obviously I already have 'ri' aliased to 'rebase -i'.

While I'm here, here are my similar git aliases for adding, diffing, etc. files using fzf:

    pick-status-files = "!f() { git status -z | xargs -0n1 | cut -c4- | fzi --print0 | xargs -0to git "$@"; }; f"
    af = !git pick-status-files add
    # and so on for diff, difftool, checkout, etc.

Oh, and 'fzi' (fuzzy-inline) is a small shell script that just calls:

    fzf --height 30% --reverse --multi "$@"

Thanks for that. I like your improvements to my idea too! :)

Yay. Btw, one more improvement. TIL xargs will skip blank inputs (depending on bsd/gnu xargs, flags etc...), so my version above:

    rif = "!f() { rev=$(git pick-commit); [[ $rev ]] && git ri $rev~; }; f"

can just be:

    rif = !git pick-commit | xargs -tI% git ri %~

That looks great. Do you have your dotfiles shared by any chance?

Sure: https://github.com/kbd/setup

Here's my git config with a bajillion aliases: https://github.com/kbd/setup/blob/master/HOME/.config/git/co...

Since git commands are one of the most common things to type, I alias all short git aliases like so: https://github.com/kbd/setup/blob/fd1f826a9365895b7a3b8b5c58...

(link to particular commit so line numbers stay correct). That way my workflow looks like:

    $ ga (fuzzy find files to add)
    $ gcm "commit message"

etc.

Can't edit this anymore.

    pick-status-files = "!f() { git status -z | xargs -0n1 | cut -c4- | fzi --print0 | xargs -0to git "$@"; }; f"

can just be:

    pick-status-files = !git status -z | xargs -0n1 | cut -c4- | fzi --print0 | xargs -0t git

I'd recently converted it from a standalone script so I just converted it verbatim into a function.

fzf is the most fun and productive tool added in my dotfiles in recent years.

Mix it with https://github.com/rupa/z or https://github.com/skywind3000/z.lua for fuzzysearch/cd

And you can do cool scripts, like:

- search your vim edited files with highlighted preview: https://github.com/BarbUk/dotfiles/blob/master/shell/complet...

- search and open your chrome history: https://github.com/BarbUk/dotfiles/blob/master/shell/complet...

>By default, Bash only records a session to the `.bash_history` file on disk when the session terminates. This means that if you crash or your session terminates improperly, you lose the history up to that point.

I kept loosing my history consistently on my new installation because of this. I usually put my laptop to sleep, and pretty much never intentionally terminate my x-session except on kernel updates. But because its a cheapo laptop with a crappy battery, I would occasionally run out of battery when asleep or it just wont wake up.

I have a fairly expensive ThinkPad and it also runs out if battery if asleep for more than about 4 days, and frequently freezes on suspend or resume when it's been both suspended and resumed docked and undocked. (Linux, of course.)

Because this comes up every so often, I'll post this link again: I wrote a little widget that stores your bash history in a sqlite db. I haven't touched it in a while because I now rely on fzf and this doesn't integrate with it, though maybe I could make it do...

https://github.com/thenewwazoo/bash-history-sqlite

This is cool, somebody recently suggested that I add history to sqlite in Oil [1]. I looked at your code but I didn't see how it integrates with bash?

Most people seem to be using the PROMPT_COMMAND hook for stuff like this -- are you using something else, or did I miss it?

https://github.com/oilshell/oil/issues/320

I will add PROMPT_COMMAND to Oil, but I'm curious if there is something else I should add to support end-user customization.

One thing that might be interesting is to support a hook for the ! syntax, so the sqlite DB could be searched. I think that would solve some of the problems you mention in the code's comments, and maybe the fzf issue. (Although that literally just occurred to me, I don't promise to do this :) )

I haven't looked at how fzf works exactly. But if you want to chat about it please chime in on Github :)

[1] Latest status: Success With the Interactive Shell http://www.oilshell.org/blog/2019/02/05.html . Oil runs a bunch of interactive programs like bash-completion, virtualenv, and git-prompt!

> This is cool, somebody recently suggested that I add history to sqlite in Oil

I think a shell with an embedded sqlite, which makes sqlite accessible to scripts as shell builtins, and in which command line history is stored in a sqlite database instead of a plain text file, would be pretty cool. You could store command history (and maybe even other stuff) in something like $HOME/.shell_db As well as just a list of lines, other useful information could be stored for each line, such as current directory, timestamp, username (maybe I su to root and am running commands as root but still going into my own account's history database), hostname (if $HOME is on NFS, putting aside the issue that databases, including sqlite, don't work well over NFS due to file locking issues), etc.

> somebody recently suggested that I add history to sqlite in Oil

what a disgusting, perturbed mind! Shell history is probably the kind of data that fits better as a text file. It is naturally textual and it can never grow too long, even if you are typing commands continuously for your entire life. Please, keep the dirty hands of sql far, far away from the shell!

I can almost see the appeal, when you think of decorating it.

Consider a table with five columns:

* uid

* pwd

* cmd

* exit-code

* datetime

Now you can `SELECT pwd,cmd WHERE exit-code != 0` to see all failed-commands.

I see your "decorated" sql table, and I raise you my text file with five columns, that you can grep and sed and awk as if there was no tomorrow.

     cat history.txt | awk '$4 != 0'

"As if there was no tomorrow" - exactly, because once you add another column suddenly $4 points to something completely different and you have to rewrite everything.

Not to mention $4 is completely arbitrary and it's impossible to tell without prior knowledge which field it's referencing.

The original query `SELECT pwd, cmd WHERE exit-code != 0` is both descriptive and won't break with future updates.

Yeah I agree. In Oil I'm hoping to have the best of both worlds. That is, no silly quoting and parsing of strings, but also being able to use plain tools like "grep", and probably SQL.

https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil

Note that R has some popular packages that allow SQL syntax over data frames.

So basically the history file could be a textual table. You could trivially import it into a database, or you can just grep it.

Ha, thanks for the feedback... Yeah I don't want the sqlite dependency by default.

The fact that somebody managed to bolt it onto bash after the fact is interesting. Oil is all about having a better/sane shell programming language, so it should have better hooks that make such customization. But yes it doesn't need to be in the core.

Since everybody has their own way of handling it, mine is

* Generate a new history file for every session (https://github.com/stilist/dotfiles/blob/master/dot_sh/sessi...)

* A 'histgrep' script that searches across all the files and highlights matches (https://github.com/stilist/dotfiles/blob/master/dot_sh/bin/e...)

I've had this in my .bashrc for years. I thought it came from this post judging from my commments, but looks different, so I wonder whether it came from HN comments when this was previously posted.

  ## Better bash history
  # Avoid duplicates
  export HISTCONTROL=ignoredups:erasedups
  # When the shell exits, append to the history file instead of overwriting it
  shopt -s histappend
  # After each command, append to the history file and reread it
  export PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND$'\n'}history -a; history -c; history -r"

I remember reading this article back when I was still using Bash. I've finally got the history I like only after switching to ZSH.

You can get decent history in ZSH just by setting the right options. If you want great history, you'll need to write a bit of code. Right now my history works as follows.

History is written to disk after every command. Up and Down keys go through the local history (from the same session). Ctrl+Up and Ctrl+Down go through the shared history (from all sessions). Ctrl+R also uses shared history (it's easy to add another binding for local history but I don't have it). Pressing Up/Ctrl+Up after typing something will go over history entries that have the matching prefix. For example, `git<Up>` will show the last command from the current session that starts with `git`. Here's my config: https://old.reddit.com/r/zsh/comments/bsa224/how_to_setup_a_....

In addition, my history is stored in Git. History from the local machine comes first, and from other machines second. This is fairly easy to do it Bash, too.

I just recently stumbled over https://spin.atomicobject.com/2016/05/28/log-bash-history/

Using PROMPT_COMMAND to log every command in a separate timestamped log file. This allows you to use a much better format than bash history has to be in.

Easily readable to see what you've been doing and extremly efficient to grep for.

Has been discussed on HN before: https://news.ycombinator.com/item?id=11806553

We have some servers where the user home directory is on nfs. To keep separate history for each server, I add the following to .profile.

export HISTFILE="${HOME}/.bash_history.$(hostname)"

that's exactly what my setting is for history, nice post. The only difference is that I unset HISTCONTROL, don't recall why I did that.

    shopt -s histappend is the default already I believe.
    shopt -s cmdhist is also the default now.
    HISTSIZE=-1 will be unlimited, which is fine for today's computers.

Meh. I prefer to leverage SQL.

I have yet to find a better way to store and search bash history than:

https://github.com/tkf/rash

More recent bash sqlite history attempts all seem to fall short than this implementation, they store less metrics or have other caveats.

Probably the most useful thing I've done regarding bash history: https://eli.thegreenplace.net/2016/persistent-history-in-bas...

I use fzf for bash history. Fuzzy history search and much better TUI.

https://github.com/junegunn/fzf

PROMPT_COMMAND is probably already used for something else by your distro.

HISTFILESIZE and HISTSIZE can be set to -1 for unlimited.

Read man bash.

HISTFILESIZE and HISTSIZE don't solve the abysmal bash history file handling when you have more than one bash open.

I would also like to know which distro clobbers user settable env variables.

can anyone please explain how BASH history on current Ubuntu, is different when logging in from different desktop machines, to the same login ? the dot-local directory or dot-gconf directory is part of this ? consternation..

Reply on news.ycombinator.com