“I have [bash] history back to ~2003” | Hacker News Reader

“I have [bash] history back to ~2003”

twitter.com

239 pointsrcthompson143 comments

  $ history | wc -l
  56201

I use a simple prompt_command hook to push my history (from all my machines) to a central MySQL database(https://github.com/fredsmith/history-db). Entries are de-duped and a tally is kept so I can do simple ad-hoc reporting:

  mysql> select * from history order by count DESC limit 10;
  +-------+---------------------+-----------+----------+-------+
  | id    | timestamp           | command   | hostname | count |
  +-------+---------------------+-----------+----------+-------+
  |     1 | 2015-09-02 19:41:45 | ls        | xicada   | 34458 |
  |    89 | 2015-09-02 14:58:33 | cd ..     | xicada   |  4332 |
  |   519 | 2015-08-24 18:38:55 | ll        | xicada   |  3229 |
  |     9 | 2015-09-02 12:37:05 | cd        | xicada   |  2250 |
  |    82 | 2015-08-28 18:27:07 | add       | xicada   |  1887 |
  |   329 | 2015-08-31 18:22:49 | gs        | xicada   |  1817 |
  |    48 | 2015-09-02 19:17:26 | git pull  | xicada   |  1696 |
  |   275 | 2015-08-24 18:23:44 | df -h     | xicada   |  1529 |
  | 15708 | 2013-10-08 01:33:33 | grid unn1 | Europa   |  1210 |
  |    44 | 2015-08-31 13:10:46 | git push  | xicada   |  1139 |
  +-------+---------------------+-----------+----------+-------+
  10 rows in set (0.02 sec)

It's pretty useful (I search through my history constantly with ctrl-R or using bash's history-search-backward hotkeys), and having a 56K line history file doesn't seem to impact history searching or shell startup any noticeable amount.

I'd be quite interested in someone provided this sort of thing as a SaaS-like drop-in to Bash and Zsh if someone wants a fun weekend project..

http://shell-sink.blogspot.com/ used to provide this service - it was what I based my project on.

I'll see what I can do for a hosted/SaaS system this weekend.

Any luck? :)

I started working on something myself--should have something out in a day or two :)

FYI this has been a side project of mine for a while. I call it Bashhub. Check it out if you're interested! Bash history in the cloud. Indexed and searchable.

https://github.com/rcaloras/bashhub-client

https://bashhub.com

https://bashhub.com/u/rccola

You could really do some interesting stuff with this. Automatically recommend aliases and show how many keystrokes you would have saved for example.

Stuff like swiftkey automatically recommends next words, it might not be that difficult to have a similar system for a shell. Sounds like a fun idea actually.

Oh, relevant XKCD. https://xkcd.com/1068/

Could be risky if the recommendations for "rm" are "-rf" and "/".

I've wanted this as well... plus keep track of how long commands took to run and (heck, why not?) output!

this must be some kind of record, in 15 words you took something great, started adding unnecessary, creeping features, and ended up with bloatware no one wants to use anymore :)

Usually this takes years!

You owe the GNU coreutils project $392

https://github.com/diafygi/gnu-pricing

This is actually awesome!

What leads you to believe OP doesn't use FreeBSD or any other Unixalike? :)

I mostly do (MacOS), but:

  $ which ls
  /usr/local/opt/coreutils/libexec/gnubin/ls

I'd say $392 seems like less than I should contribute.

Well, if it's shipped from Apple as part of their OS (I'm not sure it is from your path), then Apple would be responsible for licensing it.

Which makes me wonder, does Apple have an official stance on supporting open source they use?

It's not. Default ls on Mac OS X is /bin/ls.

I figured as much. I'm sure Apple is using some BSD version, possibly modified in-house.

OS X has been based on an amalgamation of several BSD flavors for a while now.

no, I installed gnu coreutils using homebrew. - /usr/local/opt/ is where homebrew puts installed binaries.

> This project is satire and was made during the 2015 Stupid Shit No One Needs & Terrible Ideas Hackathon.

That's good, because it would be extremely disturbing if someone from the FSF/GNU coreutils team thought charging like this was okay.

The 'announcement' for this feature is great for the ridiculous fake quotes. e.g.

"We've scaled enough. It's time to monetize." -Richard Stallman, Founder. FSF.

https://diafygi.github.io/gnu-pricing/website/

I agree with the general idea that contributing to these projects is good. However, the link is slightly misleading: "GNU programs are used billions of time every day all over the world. However, the intellectual property owners, the Free Software Foundation, don't make a cent from it. What a raw deal! This project aims to fix that by monetizing many popular command line tools that GNU maintains."

Maybe true, but keep in mind that the main coreutils maintainers (including the parent) are funded by Red Hat. There are plenty of FOSS projects that do suffer from a lack of funding; coreutils, GCC, etc are certainly not one of them. As such, if your goal is to help the FOSS community at large, you might want to consider channeling the money to other important projects.

Maybe the donation link already does such channeling; I am curious about this.

It's a joke

It's a joke that's very much not obviously a joke. This is social commentary about the state of free software and/or Hacker News, I'm just not sure what the social commentary actually is.

Seemed pretty obvious to me.

woosh! The sound of sarcasm going over your head.

This project is entirely satirical. Read through to the bottom of the README.

On one hand this really seems like overkill. On the other hand, this is incredibly awesome. Thanks for sharing :)

cray good man

  $ history |wc -l
  82230

This is just from my work machine (~4 years old), my history is not centralized but after seeing all this I think I'm going to work it into my centralized dotfiles system.

Also, I can confirm that gigantic bash histories don't affect the speed of anything.

Super useful/essential tweaks -

  export HISTCONTROL=erasedups
  export HISTSIZE="NOTHING"
  export HISTFILESIZE="NOTHING"`
  shopt -s histappend
  export PROMPT_COMMAND='history -a;echo'

erasedups will erase any earlier instance of an indentical command, keeping your history file lean. Setting HISTSIZE and HISTFILESIZE to something nonnumeric makes for no limits. histappend appends to the HISTFILE when the shell session exits. And my prompt command appends each executed command to HISTFILE immediately after it's executed. With histappend, this may be redundant. (The prompt command also gives me a blank line after each command finishes.)

That mysql setup is pretty cool too, it's a rich man's

  $ history | awk '{print $2}' | sort | uniq -c | sort -rn | head -10
  14365 vi
  11006 sudo
   6976 ls
   6898 svn
   5322 cd
   3869 cdb
   2809 svnvimdiff
   1846 ggrep
   1474 git
   1285 ssh

When I pipe history (to wc, grep) that is greater than 65K bytes on OS X, it hangs. Anyone else experience this or have an idea how to fix?

It seems all you do is git pull, push and check disk usage!! Do you really check disk usage almost as often as you do git pull and push? You do push and pull more than commits?!! I cant understand what is that you do if you dont use other commands like grep, find, vi, top as often?

I use zsh per directory history [0] as well as this cron job I have set to run hourly:

    #!/bin/bash

    cd /Users/nmjohn/.directory_history \
      && git add . \
      && git commit -m "Automatic Backup: `date`" \
      && echo "Files Modified:" \
      && git log head -1 --name-only --pretty=format:"" | grep "[^\s]" | awk '{print    "\t\t"$0}' \
      && echo ""

It's not bullet-proof, but since I never manually touch the directory it's worked without hiccup for the last ~6 months. When trying to remember the syntax to a rarely used function (or even the name itself) or coming up with a gnarly awk/sed/grep command, it's been incredibly helpful to have fairly granular search capability to go back and find them weeks later.

Also ctrl+g in a terminal toggles between per directory history and global history which makes context switching easier when switching to a project not touched for a while.

[0]: https://github.com/jimhester/per-directory-history

Great to hear you find per-directory-history useful! (I wrote it)

On the topic of the article I have per directory history going back to ~2013 when I wrote the script.

I recently set up my bash history to collect into an sqlite database. Besides the exact command it records the working directory, return value, starting time, ending time, a shell and a login session id. It doesn't handle background jobs well right now though.

Okay I'm curious as to why? I mean it's nice to have, but what are you going to do with the additional data later on?

It was mainly for an exercise to get familiar with sqlite. Maybe later very rarely I can use the session data (hunting down sessions where you only remember some irrelevant command). Getting the duration of some commands is useful like long running simulations, compilations. It's especially handy where you don't expect it to be too long and forget to time it beforehand.

Having said that I didn't use it too much yet, I have my regular history in place too.

zsh has an option to automatically print the time of commands exceeding a threshold. See here for a brief example: http://nuclearsquid.com/writings/reporttime-in-zsh/

I maintain almost all my bash history and I often use it to reconstruct a sequence of events in the past when I feel like maybe a mistake had been made.

I would love to use this! Had to grep through ancient backup diffs way too many times to resurrect that perfect command line.

I was interested myself too so had a quick search. There are a couple of implementations of the idea out there, https://code.google.com/p/advanced-shell-history/ looks the most promising of the ones I looked at. I might have to play with it later.

Do you have any links to resources you used to build it or better yet a link to your implementation?

I've been pretty happy with my bash history solution, but thinking about the return value, one big thing I think would be handy would be to have a default filter for searches that removes commands that returned an error code. They might be handy to see every once in a while, but for the most part, I wouldn't want to re-run one of them.

http://pastebin.com/zJkPW79C

I use bash-preexec [1] to hook on the even just before executing a command. I'm kind of nervous if my string quoting is safe though.

[1] https://github.com/rcaloras/bash-preexec

I guess the downside of this is that ctrl+r would only search the current day's history? Personally I at least increased the history kept with HISTFILESIZE=500000 and HISTSIZE=5000

Another issue is that it looks like would start a new history file only when a new shell is started. Some people log in fresh every day, but I've got terminal windows on my home machine that have been sitting around for many months...

I guess you'd have to set the history-size etc large enough to hopefully last for the maximum expected shell run-time.

EDIT: hmm, the man page says you can just unset HISTFILESIZE or set it to a non-numeric/negative value to avoid truncating the history file at all.

Yeah, I have this as well:

   export HISTSIZE=65535 # why not?
   export HISTFILESIZE=65535
   export HISTCONTROL=ignoredups

but now I'm wondering if I should unset `HISTFILESIZE` insteead.

It can also be nice to use HISTCONTROL=ignorespace

If you're planning on keeping a lot of history, ignorespace makes it easy to avoid inserting commands with sensitive info. A well-placed $(cat) can work too.

    $  PASSWORD=hunter2 # sensitive info doesn't go into history
    $ curl -O http://name:$PASSWORD@server/path # command with correct syntax does go into history

There is another reply defining histgrep() function. Maybe ctrl+r could be assigned somehow.

You can still use ctrl+r, but you need to provide the history file in one piece, using "history -r <file>" command. I do it in the following way, maybe it can serve you too:

  cat `find "${HOME}/.history" -type f | grep -v ".allhistory"` > ${HOME}/.history/.allhistory
  history -r ${HOME}/.history/.allhistory

I keep a pretty long bash history, but I don't have it unified among machines or anything.

Others have mentioned several handy tweaks for doing this, but I didn't see all of the ones I use so I thought I'd share them here. One of the important bits is the HISTIGNORE. I am interested in commands that I might want to search for and run again one day, so I filter out commands I consider to be clutter.

  # Configure my history preferences
    # Load history substitute into readline rather than immediately executing
    shopt -s histverify histreedit
    set histappend

    # don't put duplicate lines or lines with leading spaces in the history. See bash(1) for more options
    export HISTCONTROL=ignoreboth

    export HISTIGNORE='&:bg:fg:cd*:clear:ls:pwd:history:exit:make*:* --help:'
    export HISTTIMEFORMAT="%m/%d/%y - %H:%M:%S "

`cd` can be critical in grepping for how a file in a particular directory was made.

who actually types pwd? I've had it in my prompt since I learned about PS1.

I use `pwd -P` sometimes.

If you're going to store a lot of history, you might want something nicer than the standard ctrl-r to search it with.

[fzf] is ctrl-r on steroids.

[fzf]: https://github.com/junegunn/fzf

It's a fuzzy pattern matcher for your shell history (and lots more).

I love all of these new Go tools for managing your *nix systems.

I started using Homemaker recently https://github.com/FooSoft/homemaker which is a very clean/lightweight replacement for GNU Stow, Homesick, and other dotfile managers. The Go versions of these utilities are always simple and single-purpose.

Modern developers are returning to building Unix style systems programs. Time is being spent on that instead of the usual bulky python scripts that have been popular recently or the million personal-productivity web apps people were creating.

fzf is cool, but I am used to

ctrl-r (find something) enter (execute) ctrl-r (find something I have to change) home/end (change, then enter)

with fzf ctrl-r and selecting always gives me just a string where I then have to press enter, it just behaves like dmenu for the console basically if you look at the bash bindings, which unfortunately is a bit less flexible unless I am missing some option that would allow this

Here's something interesting - We collected bash history of around 15000 users and listed frequented used commands for fun :) http://www.webminal.org/fulc/

Kind of interesting, might be more interesting to see what the other end of the distribution looked like, i.e. the really esoteric commands that people rarely use. We all might learn something from that!

We have common mistakes list http://www.webminal.org/common_mistakes/ but didn't figure-out something like which is the least-used valid command. Good idea, we will dig deeper into their bash_history :)

I think ".." was "cd .." in dos

hmm..that's interesting,I never knew that. thanks.

Just checked the cmd prompt on Windows 7 and it isn't like that now at any rate. However cd.. (no space) works.

.. is "cd .." in fish shell

zsh and (possibly) Bash can be configured that way as well.

configured using bash alias ?

No, the `auto_cd` option, just like in zsh.

In zsh you can just use "

    alias ..='cd ..'

I have ".." as an alias for "cd .."

I also have "la" as an alias for "ls -a", and "lsa" as an alias for "ls -lah --color=auto"

Some linux distros (used to?) alias ls -a to la, which would explain why you're seeing it so often.

You are right, quick google search says 'Ubuntu 12.14' has ls,l and la. Will update our page to reflect this.

Just checked and it's still there at least in Ubuntu 14.04 LTS.

And some would also alias "ls -la" to ll. Just never the computer I'm using at the time.

yes, from time to time, I used to perform 'll' too. I guess some distro comes with that alias by default.

You should also analyse commands that were run without 'sudo'. For me, using 'apt-get install' without sudo has been the most frequent mistake I have made. Luckily, 'sudo !!' saves me every time.

thanks for the nice suggestion. Will analyse this type of common mistake too.

Doing this on my own bash_history is what prompted me to create autojump.

Just started reading about autojump on github,interesting project.

Thanks so much for creating autojump! autojump + Crl+R are saving me so much time that would be lost searching around.

There is just this impedance mismatch between our brains hazy recollection of things and the exactness that is required by computers. I find that fuzzy search combined with stats like most-commonly-used bridge this quite well!

That's like saving your toenail clippings.

I saved my milk teeth in small clear plastic bottles when I was a lad and they disappeared after a year or two from tooth decay.

Useful if you might wanna search which drug you took 15 years ago.

This guy is the inverse of me, because I have linked my .bash_history to /dev/null

Semi-serious question: do you delete all your emails after you've read them too?

No, emails are important.

I also don't run a keylogger.

haha (semi-serious laugh)

You could just disable it instead, saves some cpu cycles.

I'll do this.

Assuming I am the average Gentoo user, how many CPU hours would I save in a year? I doubt it is worth the time investment.

Why?

Why not? I mean, what's the value of knowing you typed 'ls -l' or 'ps -augx' on December 12th, 2012?

The value is in when you remember you did something before, but not exactly how. So you just grep or scan through your history, and the files you edited / commands you ran are likely to refresh your memory even if they don't exactly capture the entire workflow.

Also means you don't have to retype complicated commands too often.

Practical example: I remember I cleaned up some calibre converted epubs with perl a couple months ago, because they had a piece of text injected by some kind of generator on every page. It took me some time to figure out the exact replacement patterns, and if I were to do it again, I just need to do:

$ history 0 | grep perl ... 823 perl -0777 -i.original -pe 's#Generated by ABC Amber LIT Conv<a href="http://www.processtext.com/abclit.html" class="calibre2">erter, http://www.processtext.com/abclit.html</a>\n##g' part1.xhtml 825 perl -0777 -i.original -pe 's#Generated by ABC Amber LIT Conv<a href="http://www.processtext.com/abclit.html" class="calibre2">erter, http://www.processtext.com/abclit.html</a>\n##g' *.xhtml ...

Then inspect the history around those commands to find exactly how I got to that point.

It's moreso for when you spend time crafting the perfect command, and dont' want to repeat yourlself. Example..

find . -type d -print0 | sort -z | xargs --null -I '{}' du -sh '{}' | sort -h

Simple and clear; but history gives you the convenience to do find<C-R><CR> and having the command run with out parameter/typo verification.

To be fair, if you've built the "perfect" command, it would serve you better to put it in a alias or shell script. That way it will move with you.

It's more the value of "what command and which options did I run 8 months ago to convert data from format x to format y".

You can filter things like ls and ps.

I find that facination, care to expand on your reasons?

I assume you have the in-memory history enabled, but still

Great resource for hackers if they someday get access to your account.. but I guess you will have bigger problems than.

Yeah, I think they can do more harm with .ssh/id_rsa than with .history.

In practice I've had better luck with shell history than SSH keys when exploiting arbitrary file reads. The history tends to reveal interesting locations.

that would be why you don't put sensitive information in the command line :)

Sorry—I meant to upvote but fat-fingered a downvote.

bash history had never seemed to work for me between tmux windows. Commands are 'trapped' in their respective window and the last one to close 'wins' and writes its history over the others.

So I end-up with never-closing task-based window sessions and occasionally copy-paste commands to a safe file for future reference.

I really should investigate the root cause some day...

Your shell is likely configured to write to the file, instead of appending to it.

    shopt -s histappend

In your .bashrc should do. To have it write after every command take a look at http://unix.stackexchange.com/a/1292

Some people dislike it, but I LOVE it. I like being able to run a command, open a new terminal, and be able to control-r to find that command again. It ensures that I don't have race conditions over which terminal writes its history to disk first, as well, so I lost far fewer (zero?) history elements.

You've conflated terminal and shell there. It's an important distinction because, in fact, there's still one race condition with history files, even if you only have /one/ shell running. It's down to the Bourne Again shell not doing atomic updates when it rewrites the history file. It's one of two bugs, the other being systemd sending multiple signals in very quick succession to kill a terminal login session, that in combination cause the effect discussed at https://news.ycombinator.com/item?id=10151861 . Far from losing zero or a few history elements, it can in fact result in your losing all of your history.

http://askubuntu.com/questions/67283/is-it-possible-to-make-... might do it?

(In zsh I believe it's:

    setopt inc_append_history
    setopt share_history
)

This is one reason why this method uses a different file for every single shell session.

Put

    shopt -s histappend
    export PROMPT_COMMAND="history -a;$PROMPT_COMMAND"

in your ~/.bashrc.

Also, if you want fresh history in an already opened window (assuming you added the above), do "history -n". E.g. if you typed "longcommand" in window1, then switch to window2, do "history -n" and uparrow, and "longcommand" will be there.

Frankly the whole system is archaic.

A much better system would be to have a user daemon centralize history (among other things). The benefits would be numerous:

- Better performance

- (optional) Sync between concurrent shells

- No history overwriting problems / race conditions

In addition, this daemon could be used to enable other programs such as autojump. These programs work with the current system, but are based on a hack and as such have needlessly poor performance.

It's not just the history feature that's archaic. Bash itself is quite the abomination, also operating on bytestreams is really error-prone, and then there is the whole tty system..

As much as I like my command line, I think it would be very worthwile to rethink this whole system from scratch.

This is IMHO the real benefit to zsh over bash. You can set zsh to share history between sessions and they are interleaved based on the time the commands were run. No more closing a session in the wrong order and losing half of your history for the day.

Unfortunately, I only started saving my history somewhat recently.

    $ cat ~/var/history/bash.d/* | wc -l
    464291

    $ for year in 201{2..5} ; do
          echo -n "${year}: "
          cat ~/var/history/bash.d/${year}* | wc -l
      done
    2012: 116285
    2013: 175525
    2014: 117928
    2015: 54553

It's not a complete history, as I filter several commands with HISTIGNORE.

I was impressed that Stephen Wolfram has been recording his own keystrokes since 2002: http://blog.stephenwolfram.com/2012/03/the-personal-analytic...

Ha, quite fitting that Wolfram would be doing this.

You can start too: https://github.com/gurgeh/selfspy :)

yeah i wonder how many passwords are stored in there in free text format

   echo "Password: "; read -s pass; stuff --pass "$pass"

Where stuff takes password as an argument. Alternatively you can start your line with a space character and it isn't stored in your history.

Cool, did not know that about the leading space.

Do be aware though that this only works if HISTCONTROL is set to 'ignorespace' or 'ignoreboth' (for bash anyway).

For zsh, you need to have HIST_IGNORE_SPACE set.

The leading space behaviour is true only if HISTCONTROL="ignorespace" is set.

Did not know that, for me HISTCONTROL=ignoreboth is set in my .bashrc. I think Debian's adduser creates a default .bashrc file where this line is present, I'm not sure though.

I believe that command-line parameters are available for all to see in the output of ps, so typing parameters on the command-line is a bad idea anyway.

I wish I had bash history that long. Never seems to have the command I want when I look for it.

I think it's better to simply learn the commands than to have a tremendous amount of history logged. If you have to type a lot of command parameters a lot you can always automate that with aliases anyway.

It's not the stuff you do often that is difficult to remember.

I have <M-p> and <M-n> bound to history-search-backward and history-search-forward respectively, so

ls <M-p>

inserts the most recent command in history that begins with 'ls ' in the command prompt. I don't know why this isn't the default, as it's extremely handy. Also it works with any CLI program that uses readline.

If you want to try it out, add this to your ~/.inputrc file.

"\ep": history-search-backward

"\en": history-search-forward

I considered doing this with bash, but it was really only a marginal benefit for me. I do keep archives of all my rss/atom feeds though.

https://github.com/uxcn/scripts/blob/master/misc/bash/newsbe...

I use fzf https://github.com/junegunn/fzf combined with bash and zsh for doing some fuzzy completion of history when pressing CTRL-R. Also opens the list in a tmux pane if it's installed. Plus, it's very useful in Vim for opening files in the current buffer.

Anybody have the equivalent for fish shell? Or better yet, a config that will consolidate history across terminal tabs?

fish 2.2 ships with `history --merge`:

       · --merge immediately incorporates history changes from other sessions.
         Ordinarily fish ignores history changes from sessions started after
         the current one. This command applies those changes immediately.

I've found this tool to be super-helpful: https://github.com/dvorka/hstr

It's like ctrl-r, but it shows you many candidates at once.

Also have a look at fzf: https://github.com/junegunn/fzf

Is hostname,pid,date really enough to avoid collisions?

If avoiding them is really important, I don't think you have any choice other than to go the whole hog and basically implement half of Maildir.

Date and time to the second. For something I start interactively? Yes, it's good enough.

Well if it includes seconds, the likely hood of me typing something on a machine, switching to another and running something with the same pid within a second, is pretty low.

Mostly because I can't switch to another machine within 1s

It includes hostname too. You can only get a collision if you do all of the following within one second:

1. Start an interactive shell. 2. Enter a command you want saved. 3. Exit this shell. 4. Start another shell. 5. Be unlucky such that the new shell has the same PID as the previous one.

This is not something to worry about.

I do something like this in PowerShell too. Forgetting handy one-liners by not realizing they were important, or losing them when Windows restarted were the impetuses for me to start.

> “I have [bash] history back to ~2003”

And due to the way Bash is developed, all you need is the same Bash executable you had in 2003 to ensure all the commands actually work.

Why even store your bash history? It's like you store everything you ever said... Would you lose your precious time to go through the old stuff? I wouldn't.

ctrl r

Why wouldn't you? Do you really type long commands from scratch? I don't know how folks don't use bash reverse search on a regular basis.

I'm a computational biologist. I often need to know exactly how some piece of analysis was done months or years ago. In an ideal world all the details would be in scripts with carefully written READMEs cross-linked from my lab notebook, but it's not an ideal world and sometimes we're in a hurry. In those cases the history saves my bacon.

I'm not a computational biologist, but I've run into this once or twice. I only use the shell for quick summary statistics now though. I started doing most of my analysis in R, which made life easier on a lot of different levels.

Try binding the <UP> and <DOWN> keys (or whatever keys you want) in you ~/.inputrc to the readline commands "history-search-backward" and "history-search-forward". See bash(1) in the READLINE section details.

Instead of simply scrolling through your recent bash history when you press <UP> (which still works the same when your prompt is empty), it filters out any command that don't match the partial command you've already typed. For example, typing "cp " and then pressing up will find the last "cp" command that I typed.

Reply on news.ycombinator.com