Fixing macOS Zsh Terminal History Settings | Hacker News Reader

Fixing macOS Zsh Terminal History Settings

blog.akatz.org

132 pointsackatz64 comments

This is zsh specific, but here's something I've been burned by enough times that I specifically wanted to mention it: save your history in a file other than the default one. Why? Because if you increase the history limit, and your profile isn't sourced for any reason, the shell will read your history file and use the default history size, clobbering everything that's there. If you use a non-default history file this rogue shell (missing your settings) will dump its history into the default file and leave your custom file untouched.

I will try this, recently mine got truncated (don't know why) and it's super annoying.

If you have time machine, you may be able to recover the old history file.

I lost two history files to the ether, and repaired several more with Time Machine before I got fed up with dealing with this anymore. (I still have Time Machine enabled, but I haven't had to restore my bash history from it in years.)

For some reason, macOS will truncate your bash_history if your free space on your drive is low. No idea why they chose to truncate a small text file.

I also learned the lesson to use a different history file recently. I added `HISTSIZE=-1` to `~/.bashrc` to make the number of items stored in the history unlimited, but the history kept being truncated. The problem was that Ubuntu's default `~/.bashrc` file set `HISTSIZE` at the beginning of the file, and it had a side effect of truncating the history immediately. I had tried various methods but in the end, using a different history file felt the cleanest.

I got burned a couple of times in the past with bash-history, and go a small step beyond this, even on zsh, and added to .zshrc:

    preexec_custom_history() {
      echo "$HOSTNAME $$ $(date "+%Y-%m-%dT%H:%M:%S%z") $1" >> "~/.fullhistory"
    }
    preexec_functions+=(preexec_custom_history)

I work on a lot of shared-filesystem computers, and so it's useful to be able to filter for commands ~year ago when I know what computer I was on when I ran it. I have a fzf binding to search through this with ctrl-r, and it's otherwise easily greppable.

When I was still using bash, I had this same thing using https://github.com/rcaloras/bash-preexec .

Thanks, this looks great!

How does the history know that the prefix (hostname, PID, date) is not part of the command though?

I mean, I tested it and it works as expected (without them) but I'm really curious about how it works.

For searching/expansion, it's a constant number of fields I use my own ctrl-r handler, which just runs `cut -d" " -f 4-`. Before that, I just used the standard history and used this as a comprehensive backup - grep, and seeing everything in context was valuable for me alone.

Also I don't understand why modern shells are so conservative about the defaults. I understand that if a single history file is large, it might slow down shell startup and operation, but at the very least shells could/should do periodic history file rotation. It's very very very unlikely that the history files will be eating all the disk space.

The shell's defaults are conservative because when you change the defaults you change the setup of many existing users. So unconfigured, the shells feel much the same as they did thirty years ago.

There can be other reasons for wanting a smallish history besides fast startup like wanting low history numbers if you learnt all the ancient `!` history escapes that were the main way of using history with csh going back forty-plus years ago. This is also one of the reasons why extension theme frameworks like oh-my-zsh have become popular - much of what they do is enable bells and whistles that are features of the core shell.

Zsh doesn’t actually have a default history-file location:⁰

HISTFILE The file to save the history in when an interactive shell exits. If unset, the history is not saved.

0. https://web.archive.org/web/20221012011928id_/zsh.sourceforg...

I often have multiple concurrent sessions of terminal open, which leads to messed up history interactions. One solution I've encountered (but have not tested yet) is zsh-histdb [0]. It stores sessions' history to a SQLite db instead of a single appended file, with extra metadata about when commands were run, what session, etc. If you've already got your .zshrc file open to mess with the history settings it might be worth checking out that tool while you're at it.

0: https://github.com/larkery/zsh-histdb

If somebody is interested in macos app for storing your history (bash,zsh,fish) and syncing it with icloud, I have built an app ShellHistory, v2 is available via TestFlight if you want to try it. https://loshadki.app/blog/2022-09-22-public-beta-shellhistor... (monterey and ventura) And this is a link to v1 https://loshadki.app/shellhistory/ (big sur+)

Been using this for about a year, well worth the $12. Thanks man!

Saves me finding a command I ran but I don't remember when, or in which of my 50+ open and not committed shells. Also the exit code of every command is super helpful.

Thank you for feedback! Appreciate to see a user from HN :D

>...or in which of my 50+ open and not committed shells...

I suffered this frustration too, but adding this to my .bashrc solved it:

    #write to history file at each shell prompt
    export PROMPT_COMMAND="history -a; history -c; history -r; $PROMPT_COMMAND"

Every time a command is executed it's written to history right away. What you loose: the scroll-back buffer in each window pertaining to that session. So if you have a window running a set of commands, and you just 'up arrow' to re-run, you loose that because all history gets appended.

I too would generally have 20+ open xterm windows but this gave instant command-to-history that I needed. fzf also helps instead of relying on up-arrow.

I'm going to give this a go. That looks amazing, and it makes so much sense. I have a ton of commands around, either in my history, which isn't the best place, or in random files in different project folders. The ability to take a command and add to a notebook is a great idea.

This looks great, one of these things I always wanted to build myself so I'm glad someone did a better job and shipped it!

Thanks for sharing!

Warning while this looks sweet, but this a security breach in the making. Having mistakes done in CLI local to the machine and session has saved me many times from storing really stupid stuff in the command history.

I am not sure what safe guards are needed before I can use this. Which makes it hard to recommend it. Especially since security is not mentioned at all on that page.

A good best practice is to always change tokens that get stored in history, no matter where that history is stored (which in this case is iCloud).

IIRC, putting a space in front of the command will prevent it from appearing in history...

Just to clarify:

1. iCloud sync is optional. You can sync the history, or notebooks (or both). Or keep everything always local only. 2. ShellHistory syncs with your private iCloud account, nobody other than person having access to your AppleID (with 2nd factor auth) cannot get access to it. 3. I would say it could be even better than storing your history in a file. Considering that with the ShellHistory you have access to "Full-Text-Search" and can really quickly find any accidental leaked information. 4. In v2 I have implemented an ignore pattern with RegEx, where you can define which commands you want to be ignored from saving to database. 5. I am on purpose build the app sandboxed and distribute it via App Store. It declares what it does. I don't and would never implement any custom telemetry collection, other than what App Store provides me already.

I'd given up on shared history because it never worked the way I like. This looks very promising, thank you.

Storing history in SQLite is such a wonderful idea, I wonder why someone doesn’t just add it to readline/libedit/etc, then all apps (which use those libraries) can do it.

Although, I wonder if that might cause problems, if apps already link to SQLite and they might expect a different version, or if unstable apps might cause concurrency/corruption/etc issues with a shared per-user history DB. To avoid all that, maybe a per-user history daemon with a Unix domain socket?

The xonsh shell can use sqlite for history. Really useful, also contains runtime statistics, for instance. I used xonsh for a few weeks. It is absolutely an awesome shell, but I didn’t find the switch worth the effort personally.

Slightly silly sketch (bash):

    sqlite-utils create-table ~/commands.db commands id integer text text --pk id
    PROMPT_COMMAND="( fc -n -l -1 | perl -p -e 's/^\s+//; chomp if eof' | sqlite-utils insert --text ~/commands.db commands - & )"

It's slow, has perl & python external deps, needs a timestamp column, call subshell to avoid job control messages, ...

A nicer single "prompt command" wrapper is certainly possible though.

if we're talking about zsh history settings, I also recommend

setopt SHARE_HISTORY HIST_IGNORE_DUPS

SHARE_HISTORY will cause zsh to write to the history file after every command which means that two shells running in parallel won't override changes of each other and it will write a timestamp to the file too in order to have the history in chronological order even in light of multiple instances.

HIST_IGNORE_DUPS (or HIST_IGNORE_ALL_DUPS) will cause duplicated commands to not be written to the history file which helps with `Ctrl-R`ing

Oh! I came here to comment that this was the fix I needed: "When I have 2 shells open and close 1, the next shell I open seems to randomly choose which shell's commands to remember."

Looks like this may be what I need!

Oooh thank you!

OT but the first command I type on a new install of MacOS is

  $ chsh -s /bin/bash

because I have 20 years worth of bash scripts that I've carefully tailored to work on bash in MacOS and Linux and I don't have time to rewrite them all just because Apple decided they don't like GPLv3.

Before they changed the default the first command I had to type on a new install was

    $ chsh -s /bin/zsh

You can’t please everybody.

I recommend first installing a recent bash from macports, update /etc/shells first; then change to /opt/local/bin/bash. Unless you enjoy using an ancient bash.

Wtf, macports ? Please do your self a favor and install Nix package manager.

I did look at nix, but was less than enthusiastic about how complex[1] the software is. If I could, I would probably prefer gnu guix - but AFAIK it is unlikely to ever support running on macos - and is similarly complex (build daemon needed to run in background).

I'm also sad that pkgsrc seems to be abandoned wrt binary builds for macos - I could at least not find correct gpg keys for verifying the archives listed at: https://pkgsrc.joyent.com/install-on-macos/

Perhaps it's workable as a build-from-source package manager.

For now, it seems that macports have most of what I need - with less complexity than nix, and a better handling of upgrades than brew.

Does appear that brew moves faster, eg: https://trac.macports.org/ticket/65922 (i briefly tried a local build, but ran into dependency problems with libvterm that I didn't manage to resolve - there's a 0.3 [2] tarball, but the build system for neovim wasn't happy and kept finding 0.1.4 after it was uninstalled in favour of 0.3).

[1] https://nixos.org/manual/nix/stable/installation/installing-...

[2] https://www.leonerd.org.uk/code/libvterm/

You could still use those scripts with a shebang, no?

It's still bash 3 though, right? You're okay with that? I tend to install the newest from brew.

? Don't you just make the scripts start with `#!/user/bin/env bash`? Or do you mean aliases/functions?

I I do not do this because I have 20 years of zsh scripts.

Also, I like zsh better.

doesn’t mac show a bash deprecation warning or something else that’s obnoxious when you do that?

  export BASH_SILENCE_DEPRECATION_WARNING=1

In zsh, it should look like this when you launch bash in the shell:

$ bash

The default interactive shell is now zsh. To update your account to use zsh, please run `chsh -s /bin/zsh`. For more details, please visit https://support.apple.com/kb/HT208050. bash-3.2$

So it's easy to switch between bash and zsh. It's quite rare for me to be in situation where I need to do something that only plays nice with bash, though.

To each their own, but know that you are stuck with a really old version of bash if you do that, and it has some “interesting” bugs in it.

See something I wrote a little while ago on this topic: https://jmmv.dev/2019/11/macos-bash-baggage.html

Not necessarily if one installs bash from homebrew.

That's true, but the `chsh -s /bin/bash` that someone posted upthread does not use the homebrew version.

You can still run Bash from Zsh. Your terminal shell is irrelevant.

Especially annoying if you have scripts that are meant to be sourced by the shell i.e. the shebang doesn't help you.

I that case you could try to emulate bash before sourcing the script

I haven’t personally found any system bash scripts that don’t work on system zsh, but I don’t use many bash features because the macOS system bash was so old and outdated. YMMV of course.

I recommend not changing the default shell, but rather having Terminal.app (or whatever) start the shell of your choice (in my case: /opt/local/bin/bash). You can't trust that the files in /opt/local or /usr/local will always be there, whereas the ones bundled by macos will.

That changes your default shell, bash scripts (which should have the sheband `#!/usr/bin/env bash` will still get executed by bash normally.

> If you have a shell open and echo "test", then history | grep echo, it will return something like: 891 echo "test".

> However, if you issue 16 more unique commands, history | grep echo will no longer be able to find your 891 echo "test" entry.

On zsh, history only returns the last 16 commands, which is the default. But if you type `history 0`, it will return all commands since the zeroth index.

So if I want to search my entire history for instances of a command like this example, I don't modify anything, but just type `history 0 | grep echo` and that does what the author seems to be expecting.

Does someone knows of some config to make zsh behave similar to bash for interactive use?

I guess you'll have to elaborate on "similar". What are the differences that you want to resolve? I followed Apple's migration to Zsh and configured it to be Bash-like, at least enough for me to not notice the difference.

My google is failing, have a link to this migration document?

Git: completion of branch names

Many things are similar. What differences bother a particular user will vary. In many cases, zsh's approach is better so the value of crippling it is questionable. A common example is not splitting variables at spaces which is much better if you might have filenames with spaces in but it helps if you bother to learn to use arrays. Another example is the behaviour when wildcards don't match any files. Bash leaves the word unchanged with the wildcards in place which is less robust and fairly nasty but if you're inclined to be lazy can save on quoting.

So don't try to make zsh mimic bash, use it as-is and whenever something bothers you, try to understand both how the behaviour can be changed and consider the actual merit of the options.

Yeah need to look more into how to configure zsh, I'm accustomed to the way bash does things by default. Quickly of the top of my head are getting a pager with multiple columns of output when listing the files of dirs with many files and the readline behavior of deleting directories instead of whole paths when pressing M-backspace.

using fzf is also a good option: https://github.com/junegunn/fzf

I guess zsh is better at handling command histories of several concurrent shells than bash?

But still, you can do better. I'm using a snippet from this page: https://spin.atomicobject.com/2016/05/28/log-bash-history/

export PROMPT_COMMAND='if [ "$(id -u)" -ne 0 ]; then echo "$(date "+%Y-%m-%d %H:%M:%S") $(pwd) $(HISTTIMEFORMAT= history 1)" >> ~/.logs/bash-history-$(date "+%Y-%m-%d").log; fi'

This give an easily greppable shell history. An entry looks like this:

2022-10-13 09:14:54 /home/bhaak 1000 vim ~/.bashrc

Default settings on Monterey:

    $ grep -i HIST /etc/zshrc 
    # Save command history
    HISTFILE=${ZDOTDIR:-$HOME}/.zsh_history
    HISTSIZE=2000
    SAVEHIST=1000

I just tried this:

  >echo bollox
  >history | grep bollox

and got a long line of results, dating back to 2019 [Who knew I swore so often in my terminal!]

I don't have any of the fixes suggested in the article in my .zshrc so either it's duff info or not all OSX zsh installs are setup the same. Mind you, I'm still running OSX Mavericks so my zsh may not even have come bundled as the default shell. I may have enabled or installed it myself at a later date.

Well HISTFILESIZE is not a valid zsh variable, afaik. But the others are and they work fine. That alias does nothing besides listing history since the beginning, though it misses the first entry and should be "history 0".

Doesn’t it do something weird with the sessions/tabs as well? You write a command in one session, change to another session, grep for it and nothing gets found

Installed McFly and the history has not once been erased. Prior to that my history keeps getting truncated.

I use:

    fc -l 1 | grep foo

Reply on news.ycombinator.com