Keeping a long shell history | Hacker News Reader

Keeping a long shell history

registerspill.thorstenball.com

176 pointsingve94 comments

There's a way to configure readline (which bash and many other use) to use up arrow history, like Matlab or Julia do by default, where you hit the up arrow key to search the history for commands that start with what you've typed so far. It is a small thing, but I use it all the time.

In ~/.inputrc:

    "\e[A": history-search-backward
    "\C-p": history-search-backward
    "\e[B": history-search-forward
    "\C-n": history-search-forward

At some point in the past it took me nearly an hour trawling the internet to find this piece of code, though it's not quite identical.

  "\e[A": history-search-backward
  "\e[B": history-search-forward
  "\eOA": history-search-backward
  "\eOB": history-search-forward

The only downside ist, don't over-rely on history muscle memory! Regularly every few months I use p+arrowUp and expect to get "ping" but get the exceptional "pip" instead.

Incidentally, many distributions have this enabled, but with the PgUp keybinding instead of Up.

when I first found this years ago it changed my life. so much faster than ctrl-r to search the history... type a prefix and hit up a few times.

How is it so much faster? It's only one additional keypress, vs having to reach for the arrow keys.

Reminds me of Stephen Wolfram (Mathematica author) who has recorded every keystroke he ever typed: https://writings.stephenwolfram.com/2012/03/the-personal-ana...

Well, having read (or rather: started to read/tried to read) many of Stephen's blog posts, I would argue that not every single one of his musings should be recorded. :-)

Thats interesting to learn. I think he is on to something.

The one thing that computers are really good at is storing data (especially text) so I believe it should be a core function of any computer that it should never loose anything a user has input.

Sadly the reality is far from the truth. The number of times I have entered something into a webform or an app, only to have it disappear for some weird reason...

I find it peculiar to ask this question but then also deduplicate your history file because then you lose complete history and context. Sometimes I've had to issue series of commands to fix a particular problem (changing a kerberos password, fixing the expiry date of a gpg key, etc.) that I can backtrack through the history file to find what I did and how I resolved it, later having to do the same, and according to their settings the first instance would be removed.

Personally, I can't remember the solution to everything I do so having that deduplicated log gives me the full history of what I did and how I resolved it, amongst other uses. Sure, removing duplicate 'ls' entries is fine for example, but for every command? Not so sure.

I also don't understand why they're doing it. On my long-running full history duplicates cost me a factor of 2 in space, and presumably the same in search time (but still perceptually instant).

You know what, you're right. And it's actually something that's been bugging me for a while, I just never got around to fixing it. I'm going to update my config now. Thanks!

(Author of post)

I also like to use

  sudo chattr +a .sh_history

to ensure that my shell history file isn’t deleted.

Being append-only, it also ensures that some type of timing issue in my shell which means my HISTFILESIZE or the equivalent isn’t set to something larger than default, meaning my history isn’t deleted.

If the shell is unaware that the file is append only, does it result in appending not just the commands run from this session but undesirably also ends up appending a full copy of a lot of the previous history?

I guess it means that the open() syscall will require the O_APPEND flag. If the file is opened without that flag it will error out.

a weaker method is to set a non-default HISTFILE after HISTSIZE etc

I use zsh-histdb which stores the history in SQLite with context. Then I have few different queries/ways to interact with it. For example I have a line completion (which also shows as grayed out text while I type) based on most frequently used commands with give prefix with additional weighting depending how close in the directory tree they were used.

[0]: https://github.com/larkery/zsh-histdb

oh, incredible. this is exactly what i wished existed for years but assumed was too absurd/overkill to have ever been made, so i didn't even search for it

Good tool, but my history is so noisy I had to find another way. Lately I started to add a new habit in my shell actions.

When a command has some cognitive requirements I create a script with some ${1:-default} values and I store them all in $PATH enabled local/bin

All my scripts start with __ and then some capital letters indicating the topic.

I can copy them to new machines, which is something I would not do with my history.

Prefixing your scripts is an awesome tips. A colleague told me he prefixed all his commands with ',' and I've started to do the same. '_' might clash with 'hidden'(?) shell expansions etc.

Sadly git doesn't allow for aliases with an comma-prefix, so all git aliases starts with a dot.

You can make symlinks or shell aliases that just reference git-add et al directly.

> my history is so noisy I had to find another way

The fzf search syntax [1] can help, if you become familiar with it. It is also supported in atuin [2].

[1]: https://github.com/junegunn/fzf#search-syntax

[2]: https://docs.atuin.sh/configuration/config/#fuzzy-search-syn...

Nushell has a SQLite history mode which captures more metadata than is typical. I haven't looked into extending it, but it seems like a good setup for adding enough so that some of these cognitive requirements could be recovered from history rather than thought of up front.

Like if I tend to go down the wrong path, fix it, and then go down the right path, maybe that's a pattern that can be identified and then when I search history it finds two or three commands which together set me on the right path in the first place.

I mean, thinking ahead is always gonna be better, but I don't always.

I’ve done this enough times I chain the desired command and the cd together:

    cd ~/x/y/z && my-command

So that when I CTRL-R for my-command it automatically reminds me.

If you’re doing this for history value put it in a subshell so it doesn’t change your working directory in a confusing way:

    (cd ~/x/y/z && my-command)

Neat idea, definitely going to use this! Thanks for sharing

cwd is one of the default fields. Here's my first nushell command:

    $ history | get 0
    ╭─────────────────┬─────────────────────────────╮
    │ start_timestamp │ 2023-12-15 15:39:12.872 UTC │
    │ command         │ ls                          │
    │ cwd             │ /home/matt/src/configs      │
    │ duration        │ 31 ms                       │
    │ exit_status     │ 0                           │
    ╰─────────────────┴─────────────────────────────╯

It would take some smarts to figure out when to bind the cwd and when not to, but that would be the interesting part.

I use a similar trick but even dirtier

Whenever I'm satisfied with a pretty long command, I add a comment at the end that explain what it does (#blablabla). Makes searching sort of easier.

Prefixing these utility scripts is a nice tip, I used to do that as well.

Some time ago I found sd [1] though. It's a light wrapper around your own scripts which provides namespaces, autocompletion, custom help texts + some other QoL enhancements around that. It improves discoverability and usability a lot, very happy with it.

[1]: https://github.com/ianthehenry/sd

I do a lot of literate coding for shell work, using org-mode. I have param substitution, command chaining, textual explanation and saved output from the last time commands were run. I can save the .org files for reference, or add them directly to my blog.

> When a command has some cognitive requirements I create a script with some ${1:-default} values and I store them all in $PATH enabled local/bin

I would consider using just for this:

https://github.com/casey/just

The article briefly mentions atuin at the end. I've tried atuin but found it a little bit too heavyweight for me - instead I use zsh-histdb[0] (together with the fzf extension for it[1]) which allows you to easily answer this type of question - can highly recommend it.

0. https://github.com/larkery/zsh-histdb

1. https://github.com/m42e/zsh-histdb-fzf

I tried atuin as well and switched back to my fzf setup. I setup my history to ignore duplicates. Makes it also easier to use the up arrow key. But that can have downsides depending on the user. For me it works great for years. The only reason I don‘t share the history along with other dotfiles is the problem that some commands read API keys and token from the options. And I don‘t yet have a system for that so that. Because other means like env variables can make the it harder to repeat said command month later.

Shoutout for https://atuin.sh/, shell history across multiple devices with tons of cool features

I use—and love—atuin. It beings the power of SQLite to your shell history with sync across multiple devices. It’s simply awesome and has helped me unravel historical install and configuration details that would otherwise be lost.

Bringing this advice up again because (unlike a lot of what I say) it’s actually something that I have personal experience with and I think it will greatly improve your life: https://news.ycombinator.com/item?id=33187749

Related comment with a different solution to the same problem:

https://news.ycombinator.com/item?id=39076976

My .zsh_history doesn't go back far enough, but I can see my stack overflow history from exactly 1731 days ago (the search is surprisingly nuanced, and allows filtering by user and date range right down to the day - to see yours just replace my user id with yours in the following url):

https://stackoverflow.com/search?q=user%3A5783745+created%3A...

Looks like on that day I was having trouble aligning a navbar (no surprise), and figured out how to render a view without a layout in rails 5.

Fun fact: if you don't want to put a command in history, prefix it with a space. This works in many shells.

This feature can also be be enabled in macOS by adding the following line to ~/.zshrc:

    setopt HIST_IGNORE_SPACE

> This feature can also be be enabled in macOS by adding the following line to ~/.zshrc:

> setopt HIST_IGNORE_SPACE

Except that I think it's called `histignorespace` (I haven't caught up with when and how much zsh cares about lowercased names), I think this is just about zsh, not about macOS specifically.

And if you're using fish shell, you can start a private session with fish --private or fish -P for short

Doesn't work for bash

You may need to enable it

HISTCONTROL=ignorespace

Everyone's memory is better than mine: I can't even remember the command I typed, so I need to hunt it down from the context: current directory, date or previous commands.

I use a zsh precmd() along the lines of

  echo "$(date "+%Y-%m-%d.%H:%M:%S") $(pwd) $(fc -ln -1)" >> ~/bash/bash-history-$(date "+%Y-%m-%d").log;

Should be doable with bash's PROMPT_COMMAND if you are still on bash

We have a lot in common :-P except I'm still on bash, and do rely on PROMPT_COMMAND like you mentioned -- https://github.com/9001/asm/blob/hovudstraum/etc/profile.d/e...

Lines 11-21 can be ignored, they detect if the folder you were in got moved/deleted from another shell, to avoid the confusing behavior you get in that case.

zsh has a comprehensive hook system available for these types of tasks¹(zshaddhistory in this case). It gives you more options to control how/when the history is preserved, and allows you to be more selective in when it should be written.

Possibly worth noting for others that you'd want to think this through if you're using any of the history eliding options(hist_ignore_space² for example), as one history file may contain secrets when you're really expecting that it wouldn't.

There is also a better interface to work with hook functions through the add-zsh-hook mechanism³, which allows stacking multiple hook functions together.

¹ https://zsh.sourceforge.io/Doc/Release/Functions.html#index-...

² https://zsh.sourceforge.io/Doc/Release/Options.html#History

³ https://zsh.sourceforge.io/Doc/Release/User-Contributions.ht...

> Should be doable with bash's PROMPT_COMMAND if you are still on bash

Already done, with also PS0 and a sqlite backend: https://github.com/csdvrx/bash-timestamping-sqlite

This, plus I log all terminal output:

    script -q "$LOGDIR/$LOGTIME.log" --timing="$LOGDIR/$LOGTIME.timing"

The log files are noisy with ANSI codes, but they can be stripped, and I only refer to the logs extremely rarely (but when I do, it's extremely useful).

I'm using a snippet from this page: https://spin.atomicobject.com/2016/05/28/log-bash-history/

export PROMPT_COMMAND='if [ "$(id -u)" -ne 0 ]; then echo "$(date "+%Y-%m-%d %H:%M:%S") $(pwd) $(HISTTIMEFORMAT= history 1)" >> ~/.logs/bash-history-$(date "+%Y-%m-%d").log; fi'

This give an easily greppable shell history. An entry looks like this:

2022-10-13 09:14:54 /home/bhaak 1000 vim ~/.bashrc

With a bit of prompt hacking I send information (shell session, pwd, command, timestamp) to a fifo, which is read by small Tcl script to ingest in a sqlite database. Aturin seems nice, probably does something similar.

It's a hacky script but the idea is sound. Good thing about SQL is that I can fine tune the query before feeding to fzf. A combination of order by recency and frequency of commands with and without context (pwd) works very well for me. With proper indexing I am not close to feeling slowdown, but if that happens I can probably just export data to duckdb and use that.

I've been doing something similar for years. Originally I tried to keep everything in .bash_history, but over the years and different machines it inevitably gets wiped sooner or later.

This helps (I use bash rather than zsh):

  # Sort out history$
  export HISTIGNORE="&:ls:vi:bg:fg:history"
  export HISTCONTROL=ignoredups:ignorespace
  export HISTSIZE=100000
  export HISTFILESIZE=100000
  shopt -s histappend
  shopt -s checkwinsize

But the long-term (multi-machine) solution is to keep another append-only copy, one per machine:

  function storehist() { 
  # Bug in mac version of bash stops HISTCMD getting set in function called from PROMPT_COMMAND...
  CMDCNT=`history 1 | sed -e 's/\w* *\(.*\)/\1/'`;
  if [ "$CMDCNT" != "$LASTCMDIND" ] && [ -n "$LASTCMDIND" ]  # First time, no prev, do not log random
  then
    DATE=`date '+%Y%b%d %H%M'`;
    if [ "$LASTCMDIND" != "$CMDCNT" ]
    then
        echo "$DATE $HOSTNAME:$BASHTTY $CMDCNT" >>~/.custom_history;
    fi
    LASTCMDIND="$CMDCNT";
  fi
  LASTCMDIND="$CMDCNT";
  } 
  export PROMPT_COMMAND='storehist;  ....

A little bit over the top and crusty but it has worked like that for years so I've not updated it in a while. There is a separate sync script that puts the .custom_history from each machine / container into a folder on each machine with the orginating machine name in the file-name.

This keeps ^-R working over a long period of time, but eventually that stops being a good way of find obscure commands from long ago and then something like this works:

  cat .histories/* | grep CMD | grep WERID_ARG | grep RANDOM_RELEVANT_STR | sort -u | less

The overall cost of running some script on each prompt is negliable on a modern machine and as the OP notes the disk space does not amount to much. It becomes very useful over time. I have about 20MB of shell history that stretches back to '06. It is one of things that is easy to setup, and yet the value accrues steadily over time. Across the dozen or so machines involved I doubt that the archive would have survived long-term without the step to export it outside of the shell's real history file.

FWIW, I have

  export HISTCONTROL=ignoreboth

which I think is shorthand for ignoredups:ignorespace

my .bashrc has:

  shopt -s histappend
  export HISTFILESIZE=1000000
  export HISTSIZE=1000000
  export HISTTIMEFORMAT="%m/%d/%Y %T "
  export HISTIGNORE="exec env*:history*"
  export HISTCONTROL="ignoreboth"

It seems my history preceeds "recorded time", because timestamps start in 2016, which is probably when I discovered HISTTIMEFORMAT. 110k commands btw

the "exec env*" is because emacs tramp uses that when remotely logging into a machine and fill the logs with junk.

The fist thing I do on any machine is to set HISTSIZE and HISTFILESIZE to large numbers. It is wonderful to refer back to how you set up a machine from scratch for instance.

How did I set up this. what did I set up. What packages did I install.

I've always kind of wanted to create some sort of "annotated history" program. The idea would be, to ask you what you are doing (maybe in a separate window to the side) so you could properly document something fiddly.

Perhaps an edge case, but what about all the commands you mistype or use the wrong flags for etc. If I want to recall a niche or complex command I used months ago I probably won't remember which of the 10 results that pop up was the right one.

Usually the most recent one is the one with all the kinks ironed out.

You can set your shell not to store commands that returned an error exit code.

learning about ctrl+r on zsh feels like magic for me -- I'm not a heavy terminal user but I am pretty comfortable with it, and like the author, I have a lot of common commands with specific flags I need to add depending on if I'm doing a push to a local repo or a public one, etc.

I like features like this that build off of how _i_ use the shell, and it's up to me to figure out how I want to use it, and it's so simple but it makes using the shell all that much more efficient and it's easier for me to express my intent to the shell. just very cool, like learning a bit of the coreutils to make every day stuff in system administration easier.

This reminds me of people who have 9999 tabs open instead of using bookmarks.

and then freak out when their browser crashes and trashes the session. If you're that dependent on it you should probably add one of the many tab extensions that will save your list of open tabs to a backup every few minutes.

I do this. I use sidebery which creates snapshots of all open tabs. I usually have between 100 and ~600 tabs open. Once a month I go through the list and close those that are not relevant anymore (can easily select and close multiple tabs in sidebery).

It works pretty well for me, it often reminds me of things I was working on but haven't yet finished.

Since I have so many tabs, I also created a firefox extension to pin tabs on the right, so that they are close to the "new tab" button.

I think this is more analogous to automatic bookmarks.

open tabs is more like a cluttered desk.

You know, I would love a browsing history (as a log file). I think it would be helpful and useful.

In my experience, the default shell history settings don't work well with multi-shell environments such as tmux. What I have done is in ~/.bash_profile, I create a subdir of ~/.historydirs and put in each of my history files (HISTFILE, LESSHISTFILE, PY_HISTFILE, etc.) which will be unique for a given tmux pane. This means that I can not only search for an exact command, but I can also get useful contextual information related (e.g. What folder did I last cd to?)

Just a general recommendation from someone who keeps a long history: if your shell loads a histfile, don't let it grow unbounded.

It isn't massive, but there's real shell startup latency associated with loading lines from your histfile. A few hundred isn't really noteworth, but tens and eventually hundreds of thousands of lines can add up.

(In my case, i store them in a database and I synthesize histfiles from the db. Keeping the permanent storage in a separate file also probably works.)

I think histappend might help

Even if it were slower, I also think the overhead of something like that is more than made up for by the help it provides.

Totally agree with this. I use https://github.com/larkery/zsh-histdb slightly modified to work more smoothly for me. If I remember correctly, I tried Atuin but it messed up multi-line commands. Zsh-histdb handles them well.

Can fzf or the shell history file answer this question? zsh records timestamps and runtime if extended history is enabled. bash needs HISTTIMEFORMAT to be configured properly. I haven't used fzf so not sure if it can search by timestamp, I only see a line number for the matching command (presumably) in the gif.

FWiW the authors setup:

    Here’s the ZSH configuration I used for many, many years:

possibly cannot fully answer the question as there are settings to scrub duplicate commands ... meaning the target day will show commands unique to that day but not necessarily all commands from that day.

It's a small quibble but potentially an issue for those that really need to Indiana Jones their command history for some specific clue.

I agree, especially if the focus is not on the command itself but the date. Atuin has a CLI switch for filtering command history by date, but I don't see any support in the ctrl-r UI eg. with an operator like "before:" or similar.

In my own case large clumps of similar commands hinge upon what was the CD <current directory> at the time ... and the command most likely in many cases to be scrubbed as duplicate is a cd <one of 50 common working base directories>.

This leads to "I can see I tidied up a bunch of project files first monday of last month ... but which project was that??"

A nice thing that zsh (and bash-preexec) has is that you can add arbitrary functions to be called on command execution.

I work at a large facility with a shared filesystem and many computers, so often I need to answer questions like “What did I run the last time I was sitting at computer X”. I log: command, timestamp, PID (to separate terminal streams), hostname and cwd.

I’ve set up ctrl-r to search this “.fullhistory” file instead.

This sounds interesting! Can you elaborate further?

The function:

    if [[ -n ${ZSH_VERSION-} ]]; then
      # ZSH doesn't split command over multiple variables
      preexec_custom_history() {
        echo "$HOSTNAME:\"$PWD\" $$ $(date "+%Y-%m-%dT%H:%M:%S%z") $1" >> "$CUSTOM_HISTORY_FILE"
      }
    else
      preexec_custom_history() {
        echo "$HOSTNAME:\"$PWD\" $$ $(date "+%Y-%m-%dT%H:%M:%S%z") $*" >> "$CUSTOM_HISTORY_FILE"
      }
    fi
    # Add it to the array of functions to be invoked each time.
    preexec_functions+=(preexec_custom_history)

It's harder to search for multiline but I haven't gotten to fixing that yet. I've thought about putting it into sqlite (pwd makes the file large) but since everything is on network filesystems I don't think that's a place sqlite has any guarantees; screwing up a single line of a bare log doesn't break anything.

Edit: The fzf is kind of hacked together but is here: https://pastebin.com/sTxsvZAf . It has a few bugs but nothing annoying enough to have made me fix it.

Atuin (the tool mentioned later in the post) stores timestamps and CWDs for commands in a database, which is probably why the title is what it is.

I've been using it for 6+ months and it's actually good. I also like having an sqlite database with my commands so I can search it even more freeform if I want.

A little known secret is that you can actually search not only backward (ctrl+r), but also forward (ctrl+s). The problem is ctrl+s is bound by default to (obscure terminal thing) that stops your terminal from outputting anything. This can be disabled with stty stop ^-, and then ctrl+s will work as intended.

I recently reinstalled my personal laptop to distro hop. the only two files I migrated over were my pgp key and .bash_history, the bash history is around 8000 lines I don’t want to loose.

sometimes I ‘sort | uniq’ the bash history to filter about duplicates

I often use diff in this way to compare command outputs: diff <(somecmd | sort) <(othercmd | sort)

I find there are maybe about 10 commands I use regularly that I care about being in history. The rest I never search history for.

So a couple of thousand seems to do well by me.

My setup is so sporadic it would be hard to remember to save shell history. Is there any way to upload this to a server automatically across multiple computers?

Private git repo?

I’ve not used it but Atuin has good reputation and support uploading encrypted history to their (or your) servers.

https://atuin.sh/

Installed it on my machines, we'll see how it works out. Thank you!

I use a private github for my dotfiles, run a cron job to back them up nightly

PowerShell here, I use this:

    $MaximumHistoryCount = 9999

The other day there was another post about a bash history in a db, years worth and searchable and available on all machines.

I do not get this. I have over 30 years certainly cobbled some complicated commands, but I never need an ancient one.

If it's complicated and used more than one day, it's a script.

Complicated or not and used a long time ago, it's no longer valid.

These days "so old it's no longer valid" is barely a year. Back in the sco Unix days you could have 10 year old commands that are still valid references, but Linux never had that except for trivial things.

There are exceptions but they are exceptions and so by definition not reasonable to focus on.

I think that building essentially mini apps right on the command line and then carrying around years of shell history to reuse them is solving a problem the wrong way. A stupid and backwards way.

It's losing sight of what the problem actually is and like growing one weird finger muscle to the size of a leg instead of realizing maybe you're not supposed to still be using a finger for that job.

It's useful for several other things. Automatic logging is one. Another: I compute the commands run most often (candidates for automation).

> These days "so old it's no longer valid" is barely a year

I don't think this is true (generally). At my work, we use old boring technology for most things (as we should). I don't see how it could be true enough to warrant not keeping a command line log.

The author mentioned this objection and suggested that since the cost of keeping a long history is so low, there’s really no reason not to. Additionally, you may not know a command will be useful in the future at the time you execute it.

I suppose everyone’s workflows are different though. I keep a long history but rarely use it.

I do a lot of analyses as a series of shell commands. Sometimes the command is complex, other times it isn't but the specific arguments I'm using matter a lot. When I need to revisit an old analysis being able to see the commands I was running is incredibly useful for picking up where I left off or answering people's questions about exactly what I did.

These are rarely commands so complex that they represent serious programming effort, but that's not why I want to save them.

(My approach: https://www.jefftk.com/p/you-should-be-logging-shell-history)

interesting, thank you

> If it's complicated and used more than one day, it's a script.

That was a great solution prior to tools like fzf. It's more overhead creating such a script than using the OP's solution.

In ye old days, I would sort/organize bookmarks. Bookmark search solved that problem. And for a long while, Google solved the problem of even having bookmarks.

Same idea here. Why go through the trouble of making a script (and remembering its name), when you can find the command within seconds?

Predictive search/auto completion is incredibly useful.

In the browser context, auto completion search from history in the search/URL bar hugely reduces the need to even make bookmarks for top level sites. It's just much easier to type the first few characters of the site I want then hit enter.

Exactly my point.

I have a very common use case. I have had old machines that I have had to recreate or upgrade from scrach, and the shell history is instrumental in supporting this.

just a simple example:

  history | grep pacman

Did somebody say atuin! https://atuin.sh/

Yes, the article did.

Reply on news.ycombinator.com