Fly through your shell history | Hacker News Reader

Fly through your shell history

github.com

244 pointslovestaco96 comments

I wonder if having the search use context like cwd is a win in practice or not. I’ll have to try this out to see.

I’ve been using `history | grep` since forever (usually `history | grep foo | tail -n 5` which has been a single-character alias for more than 30 years). Hard to beat. Since history gives you the line number you can just `!12345` — no need to copy/paste, and works through an ssh connection!

> I’ve been using `history | grep` since forever... which has been a single-character alias for more than 30 years...

I have "history | grep" basically aliased to "h?"

Ok? Sounds like the person you're replying to said they did too? Why are we ending our statements in question marks.

My understanding is that the question mark is part of the alias name.

Oh that makes sense. I never am completely sure how to deal with the interaction of quote marks at the intersection of code and English

Yup: that's why I put in quotes... I'm literally using "h?" as a Bash alias, as GP said.

It may be a bit weird to use a question mark as part of a Bash alias that said: sorry for the confusion!

For what it's worth, I like to use backticks in these situations (e.g. `h?`). I think it reduces ambiguity a bit.

I do this too although it can add ambiguity when writing about the shell!

Why are we ending our questions with full stops!

Great question?

Because we sheeple kowtow to the periodarchy..

I’ve decided to end sentances with two to try to avoid their ire..

It definitely is (IME, and YMMV of course). Fish shell and the similar zsh-autosuggestion plugin work with directory context, and it almost always suggests the intended autocompletion (not sure why this McFly thingie needs a neural network for this when a per-directory most-recently-used pattern matcher already works perfectly).

I’m surprised that it is useful because I often want to use the same commands in different directories!

But just to be clear: I didn’t at all write “surprised” as some polite euphemism for “this is dumb”. Clearly people have found it useful for a while and it’s harmless easy for me to collect some empirical data, so I just installed it. Might turn out to be a big win.

In Fish you can still do that. The cwd is a hint to the candidate sorting algorithm, but not a strict requirement.

If anything is going to beat fish, it's probably something with a neural network. I wonder if it is actually better.

Just curious, how do you grep for foo in the middle of an alias like that? I usually use a function for something like this, or are you using alias colloquially?

Probably a bash function rather than an alias.

Yeah, lazy imprecision on my part. A function in my .bashrc .

    $ cat ~/bin/f
    #!/bin/bash
    if [ "$1" != "" ] ; then
      find . -type f | grep -v '\.git' | grep -i "$1"
    else
      find . -type f | grep -v '\.git'
    fi

Piping `find` through `grep` like that is inefficient, find has robust filtering capabilities you can use without the overhead of spawning subshells with pipes and calling another binary like grep.

    find . -type f -not -path '*/.git/*' -name "*$1*"

Hrm... `f foo` vs. `.............................................foo..`

I just fzf through history, works like a charm

This is by far my favorite fzf feature, I barely use all the other (awesome) things it can do, but I probably use the history thing 100 times a day. It's funny because it's replaced some of what I used to do with aliases and shell config - why remember a command when I can just fzf and pull it up effortlessly ;)

I find it very useful. I made a tool similar to mcfly (before knowing it existed) and use this workflow (`--here`) constantly. Also hostname context and shell session can be useful at times, too, to reconstruct something in the past.

https://github.com/chipturner/pxhist

If any one has tried this, can you confirm whether it copes with having multiple shells overwriting the history files

There is nothing more disappointing than not being able to find the cool command line I used a week ago because the shell I opened briefly and forgot about has overwritten it - maybe that's my poor setup though

This is my least favourite thing about shell history as well. It fails at being shell history log, because it's dropping more than it retains.

I hate that too but it’s worth noting that this is a bashism. It’s not a problem with some other alternative shells.

Yep, zsh, and most likely fish and other shells manage to keep all your commands in the history, no matter how many sessions you open.

I find this can be a bit messy because VS code runs lots of commands in the background that pollute the history with extremely verbose commands.

Why does it run them in interactive shells? Not surpising given it’s Microsoft, but still worth reporting an issue.

I can’t speak for VSC specifically (I don’t like their terminal emulator so don’t have enough experience using it) but I do know some other applications “paste” (for want a better word) commands into the script interactive shell because they purpose of the shell being open is to provide a REPL and the GUI application just provides a few useful graphical hooks to it.

Regarding the history problem, there’s likely a flag VSC can pass to Zsh to disable writing to history for that session.

Edit: looks like there are several ways: https://unix.stackexchange.com/questions/692913/temporarily-...

Bash was written when disks were a few megabytes. It’s trivial to tell it to keep an infinite log, as others have commented.

The issue isn’t the length of the history. It’s that the history is written to at the end of the bash session. So if you have multiple sessions open then one session with overwrite another session log.

Even if your point was right though, I’d argue that because Bash is still being maintained, any defaults originating from the 80s should be updated. Yes, I know there are low footprint environments out there that still have writable storage in the region of a few megabytes (or even less), but we have other shells for those use cases.

Changing defaults can have unexpected consequences, especially ones that have storage implication and especially in programs that are so tied to scripting.

I agree that the default is unfortunate but the points above seem more important.

Changing bash history retention wouldn’t affect scripts. Scripts don’t write to the history log.

And bash has already changed numerous features over the years.

shopt -s histappend together with HISTSIZE=99999 and HISTFILESIZE=99999 into your .bashrc should do the trick.

Yes, mcfly can handle that.

I've been using it for the last couple of years. My history file has around a quarter million entries so far.

JFC

If you use bash I'm assuming you don't have ...

export HISTCONTROL=ignoreboth:erasedups

Somewhere in your rc files.

If you do, ehe, that's some serious variety of work on the CLI !

I didn't mean the bash history file but the mcfly internal one.

mcfly doesn't remove dups and also stores additional info about the executed commands like pwd, timestamp and return code.

Now I'm wondering if I overuse / abuse my CLI. I don't find this to be a surprising number of deduped entries in the least.

I use this in my bash profile:

  export HISTSIZE=65535
  shopt -s histappend
  PROMPT_COMMAND="history -a;$PROMPT_COMMAND"

I miss it though on other computers and suffer until I set it up on new ones.

> 65535

Why limit yourself to 16bit?

Just a big enough number and I punish myself as I would like to create my own shell/terminal thing and use that instead.

I hope you get to it eventually, but if you do, consider 32 bits instead.

Computers are fast, memory is cheap and 16 bits is not that big. My .bash_history is clocking at 55814 now with dedup on. Granted, it's beeing going for years, but still, no reason to punish yourself.

>>>55814 now with dedup on. Granted, it's beeing going for years

As you can see, Mr. Anderson, we've been keeping our eye on you for some time now

I wish

Important! My history file on my laptop is up to 200,342.

With dedup on? That’s some serious typing.

Dedup off, since I use my history to reconstruct things I've done and for that I want the full sequence of commands.

Checking now it's 104,545 unique commands. It goes back to April 2012.

Note that this is just my laptop; things I've run on my work computer or when ssh'd somewhere else aren't included.

The last time I attempted this, it caused going back through the history in a separate shell to show entries from all other existing shells in order. Since I use each shell as separate workspaces, the clashing was unbearable. Now, I do:

    # Undocumented feature which sets the size to "unlimited".
    # http://stackoverflow.com/questions/9457233/unlimited-bash-history
    export HISTFILESIZE=
    export HISTSIZE=

and just depend on exiting my shells to store data long term.

If you're using bash 4.3 (released 2014) or higher you could convert this from using an "undocumented feature" to use the documented feature that does the same...

> "Numeric values less than zero result in every command being saved on the history list (there is no limit)."

  export HISTFILESIZE=-1
  export HISTSIZE=-1

    promptFunc() {
        echo "$(date +%Y-%m-%d--%H-%M-%S) $(hostname) $PWD $(history 1)" \
          >> ~/.full_history
    }
    PROMPT_COMMAND=promptFunc

Or, in zsh:

    precmd() {
        echo "$(date +%Y-%m-%d--%H-%M-%S) $(hostname) $PWD $(history -1)" \
          >> ~/.full_history
    }

How does it compare to https://atuin.sh/ ?

Some discussion about cross pollination https://github.com/cantino/mcfly/issues/373

Atuin has encrypted sync, that'll be a big difference.

I've used mcfly for years and I appreciate the hard work cantino has put into it, but mcfly is currently looking for a maintainer while the maintainer of Atuin has recently committed to doing so full time (https://ellie.wtf/posts/i-quit-my-job-to-work-full-time-on-m...). That may make a difference to some people considering which one to pick.

Both are implemented in Rust and use SQLite as storage, so both should be equally performant in terms of startup latency and storage capacity.

The ctrl+R UI of each is different but that comes down to personal taste.

I just tried to use atuin... I have to login, but I don't want to :( I tried to run a search locally... but I have to run a PostgreSQL server??? Why isn't it just allowing embedded DB like mcfly? I know this isn't the place for support, but I give up trying to get it to work :(

You don’t have to! Only if you want to sync your history. For pure offline usage that’s not required

I’ll clarify in the docs

> I just tried to use atuin... I have to login, but I don't want to

It didn't ask me to login...

> I tried to run a search locally... but I have to run a PostgreSQL server

PostgreSQL is for the server? You don't need a server. It uses sqlite, verifiable by running

$ sqlite3 ~/.local/share/atuin/history.db $ SELECT * FROM history LIMIT 10;

Did you actually try it? I don't know where you got all this from.

I'll also plug my project [0] as another alternative that supports syncing (similar to Autumn) and also has a number of powerful customization features (e.g. custom columns to collect arbitrary metadata with each command, like the git remote) and an AI shell command generator.

[0]: https://github.com/ddworken/hishtory

To me, this is the biggest difference:

> The key feature of McFly is smart command prioritization powered by a small neural network that runs in real time. The goal is for the command you want to run to always be one of the top suggestions.

This looks really nice. You say you record the number of times a command was executed. I'm currently using FZF for ctrl-r search, and it's amazing, but one thing that bothers me is that I have some commands that I search for and use quite often, but then I don't need them for a while, they vanish from my history, and then I need them again.

Would the internal database of McFly keep often searched for commands from the past, even though they're out of the bash history?

If you set your HISTFILESIZE to -1 Bash will keep your history indefinitely.

> Non-numeric values and numeric values less than zero inhibit truncation.

I believe McFly hangs on to all history in the database, though you can limit the UI to only return the most recent entries (MCFLY_HISTORY_LIMIT).

One side note -- I made a small tool to integrate the McFly database with the fzf UI, which might be of interest to you [1]. Overall I've been a happy user of McFly for a few years, and it's 100% an upgrade over bash's command history (properly storing commands from multiple terminal windows; collecting working directory + exit status info, etc.)

[1] https://github.com/bnprks/mcfly-fzf

I've been using Mcfly, Autojump (j), and Ag (a) for so long I'm basically useless on systems where I don't have them. Wouldn't change it for the world.

This looks good and I'm sure it was created to scratch a particular itch but a simple approach to this (in bash) is changing the $HISTFILE depending on 'context'.

I personally have a custom shell function inspired by virtualenvwrapper's 'workon' that sets the stage, so to speak, based on its arguments. One of the things it does is export various env variables including HISTFILE.

For me this is a more flexible way to introduce customisation (for instance one of the last thing my function does is look for a project specific function (name of which is env derived) and execute it ...and then it's turtules all the way down )

For any of the shell history solutions out there (myfly, atuin, fzf) is there a way to filter out commands that failed or had incorrect syntax? In my history there are ... a lot of those. Not that useful to have them hanging around.

Was expecting a 3d flying simulation featuring shell history as content. Oh well.

It made me think of the Jurassic park quote, once again [1].

[1]: https://www.siliconbunny.com/fsn-the-irix-3d-file-system-too...

Same. I thought we were finally making Hackers (the movie) a reality.

I hoped for literal flying, like in the Jurassic Park operating system:

https://m.youtube.com/watch?v=JOeY07qKU9c

I've been using fzf ctrl+r for a while now, but i'll give ethis a try. Also didn't realize homebrew for linux exists!

As a non-Mac user I really want to know why one would prefer Homebrew on Linux.

My assumption is OS package manager would always have more packages compared to Homebrew, and I don't know if Homebrew on Linux even supports GUI packages or not.

I was today years old when I learned about CTRL+R. I've always just done history | grep

Also, I use zsh which filters history when using up/down arrows to whatever you typed. So you can do "git" [up] [up] [up] to cycle through all recent git commands which is super useful.

now try it with fzf. You will have two birthdays

+1 for fzf, never leave home without it

Surprised to see such heavy reliance on shell history. I view shell history as a liability. Too much chance of capturing passwords or other secrets and shell history files are the first place an intruder will look for those.

My shell history has a limit of 25 commands and is not saved after logout. I very rarely find that to be too short. In almost every case, if I want to repeat a command it's either the previous one or no more than a few prior than that.

Anything I need to do repeatedly I put in an alias or script or function.

For the last 5 years, after being fed up with losing or overwriting my bash history, I've installed a daemon on all of my machines that tails ~/.bash_history and uploads every new line to a hosted time series database. I wrote some simple tools to query and search through this DB for commands. It's a super simple solution but works wonders, especially for digging up a command I ran on a different machine.

What neural network is being used I think it would be better if you told us the neural network that is doing the prediction.

It is a custom pretrained NN with very few nodes, the full source code is here: https://github.com/cantino/mcfly/blob/master/src/network.rs

I use https://github.com/atuinsh/atuin

As a back up to my large shell history, and using atuin w a self hosted atuin server), on most of my frequently used machines, I have a spunkUF running anyway, so I also have it logging/indexing .bash_history (to my offsite spunk server.). (so I have powerful search and timestamps for any of my commands, forever)

Lost me at 'neural network'

It's actually quite neat, doesn't have to do anything with the current AI hype: https://github.com/cantino/mcfly/blob/master/src/network.rs Just a few nodes, hardcoded weights. I wonder how it was trained.

I'm enjoying this immensely. Thanks a bunch.

Love the name :)

loving the name of this, super useful thx

I find it annoying that the tl;dr summarizes two sentences.

For something more rustic, an .inputrc file with this:

    # history-search-backward / history-search-forward bound to the up and down arrows
    "\e[A": history-search-backward
    "\e[B": history-search-forward
    "\e[C": forward-char
    "\e[D": backward-char
    set show-all-if-ambiguous on
    set completion-ignore-case on

Will search through anything matching what you've already entered. Type something like "gr" and scroll up or down to see all your previously used grep commands. Saw this long ago on here and it's on every machine I use now, can't live without it. Thankyou to whoever originally posted it.

Magic! Thank you, now in my .bashrc files.

This is something I can't live without either. I'd recommend people try this before trying to incorporate fzf, ctrl+r, etc into their workflows since it's simpler and probably covers 99% of your needs.

While I agree one should have these keybindings, fzf + ctrl+r is a lot more powerful, and many a time I was on a system with those keybindings, and it would suck I did not have ctrl+r + fz. I'd say it covers perhaps 70-80% - not 99%.

does this work on a mac?

Just tested, it does

I just use this for quick history search of typically my last few commands in .zshrc:

    # tsch-style history search esc-n/esc-p
    bindkey '\ep' history-beginning-search-backward
    bindkey '\en' history-beginning-search-forward

If I need substrings or searching for something further back in time I just use `history -99999 | rg`

Tried ctrl-r bash style for about a month back when I was switching off of tcsh and hated it.

How does this differ from https://github.com/zsh-users/zsh-history-substring-search ? Except that yours seems to be built-in and zsh-history-substring-search is ~800 lines of zsh

Reply on news.ycombinator.com