Hacker News Reader

Comment by mmh0000 | Hacker News Reader

I read this article in horror. Even after he said he kept /only/ the last 9800 commands.

  ⟩ history | wc -l
  170919

I have (deduplicated) shell history with timestamps going back years. A little bit of CTRL+R and `history | grep …` I never have to remember how I did some arcane magic in the past, just that I did.

View on HN

Maybe I'm a hoarder. My timestamped histories from four machines in four different locations added up to 898028 lines --- I kept most of my shell commands, including errors and iterative attempts to get a good command, since early 2009 in the servers that matter the most to me. I use named shells with different histories each (roughly one to four shells per project), so the largest single shell history in these logs is only 43424 lines (21712 commands). The average across all shells is a bit over 75 logged commands per day, which doesn't sound as bad as the total line count in these histories. Despite the advent of LLMs I still need a lot of internal esoteric hacks that are relatively easy to find in these histories. Perhaps I should switch to using a centralized db that keeps the outputs as well and use them to finetune LLMs, but the ease of simple unix/Emacs tools operating with standard/simple history files always felt attractive.

Same here. Also included the host, timestamp, user, and current directory. Unifying history across multiple machines is very pleasant when you know you did something, but don’t even remember where you did it.

What tool do you use for that?

I bet it was “bash” (with sed and rsync)

You forgot jq. The shell history is stored as json lines. Makes for easy grepping too.

I mean how do you unify it? Is there a tool that will sync/unify the history for you?

There is - and was featured here a couple of times. atuin dot sh. But it uses sqlite under the hood afaik

How long does it seem to take to open a new terminal and reach a working shell?

Your hardware might be fast enough that it's negligible, but several years ago (maybe 2018-19ish?) I noticed my shell startup was getting sluggish and traced a large fraction of it back to the time bash took to load many lines into history.

I still like keeping it all, so I wrote something to dump it into a database as I go but then truncate working histories to something shorter (500, I think).

Instantaneous. bash might just be slow here; I'm using zsh.

  $ time zsh -i -c 'exit'
  zsh -i -c 'exit'  0.03s user 0.02s system 100% cpu 0.055 total

Are you sure it's loading history in there? My .zsh_history isn't completely empty, and when I run the same with 'history' swapped for 'exit' it doesn't print anything. (But this might have something to do with macOS default shell profile stuff.)

Not confident, no. But even running an interactive zsh manually and exiting as fast as humanly possible is within the same ballpark, modulo human reaction times.

  $ time zsh
  $ ^D
  zsh  0.14s user 0.04s system 80% cpu 0.219 total

It's just not an obstacle to spawning a new shell and using it.

A way to ensure the zsh invocation behaves as a typical interactive shell is:

  time zsh --login -c 'logout'

Note use of logout instead of exit. In this context, logout ensures whatever combination of flags used results in a login shell.

See zshbuiltins(1).

On par with the first time.

  $ time zsh --login -c 'logout'
  zsh --login -c 'logout'  0.03s user 0.01s system 99% cpu 0.037 total

Whenever I've had noticeably slow zsh startup times in the past, it was almost always some plugin/extension doing something very dumb (e.g. stuff like full 'git status' in a large repo -- just takes time); not the history management.

nvm plugin used to be really bad!

i think you can debug-trace loading time putting some commands on your zshconfig

> On par with the first time.

This was my experience as well. The --login flag was recommended in order to address concerns raised by @abathur.

> Whenever I've had noticeably slow zsh startup times in the past, it was almost always some plugin/extension doing something very dumb (e.g. stuff like full 'git status' in a large repo -- just takes time); not the history management.

Great point.

Unfortunately, my limited research suggests tracking down which plugin or extension is the root cause is a manual effort starting with the contents of the canonical zsh initialization files (often named .zlogin, .zprofile, .zshenv, and .zshrc).

Is .14s noticeable? That’s more than a couple frames at 60fps.

I mean, this might seem really bizarre to worry about, but some folks use tiling window managers and just expect to immediately start typing once the “new terminal” key is hit…

Actually this has been slightly annoying me lately after making my system pretty. I added some fancy compositor stuff and now if I do “open terminal,” “exit” the computer complains that it doesn’t know what xit means. Not a big enough problem to fix though.

Edit: huh, actually playing with it this seems to be 99% the fault of the terminal emulator anyway, not the shell or my silly special effects.

The 0.14s includes me noticing zsh has started and typing ^D.

This is why I switched to `fish`. Customizing zsh to achieve feature parity with fish (along with the tide prompt) made zsh veryyyy slow. It's entirely possible that there's some sort of optimized loader for zsh out there that ameliorates this, but I just couldn't be bothered.

> Is .14s noticeable?

Unless the system is under memory pressure, most shell initialization will read from in-memory OS file caches and not be noticeable as you note.

Where significant delays are often seen is when a seemingly innocuous extension uses network-based or some other heavy file system I/O commands (such as a "find $HOME -type f" type of thing).

Even under memory pressure, it's not a ton of SSD IO time (e.g., mine is 5 MB).

Actually, this seems like an interesting question. I don’t really see any reason why a shell must load the history file before the user starts typing. I mean, I usually don’t ^r immediately, so to speed things up it could:

* Load asynchronously

* Only have a barrier if I reverse-search.

Might be over-engineered, though.

zsh may very well be doing this, especially for any de-duplication index.

    time zsh -i -c 'exit'
    zsh -i -c 'exit'  0.33s user 0.31s system 97% cpu 0.658 total

Yeah... I've definitely got something going on here.

Just put them in files, like month per machine/mpount per shell (bash, csh, etc). Then the shell will only load the current month's history. You can use grep of the files and easily narrow down what to search in.

Sure! Something date-based is a simple way to handle perpetual storage while keeping the active history set from over-growing.

I did anything at all because the default shell profiles in macOS can cause history loss, and I'd found my last straw. I put them in sqlite because--if I was bothering to build something bespoke--I wanted to track more command-time context to build tooling around later. (Including to play with some ~curation ideas.)

I might have decided to just use Atuin if it existed at the time, but I did it three years and change before its first release. (There were maybe 5 or 6 barely-used public examples of this idea on GH at the time, but none tracked everything I wanted among other issues.)

The only time I've ever had shell startup time be significant is when loading fancy shell rc stuff (oh-my-zsh is the most infamous but there are many others)

Unfortunately I've hit some obscure bug with exactly when `HISTFILESIZE` is applied, and had some shells truncate all my history undesirably. There are also problems with reading history if the timestamp format changes.

Yeah. On one host:

  wc -l .zhistory
  121104 .zhistory

Reply on news.ycombinator.com