Clearly I am missing some setting. On zsh on macOS and I tried what you just said and it doesn't work like this
type eg "rg" then up or down arrow, and the history shown is filtered by that rg prefix.
bindkey '^[[A' history-search-backward
bindkey '^[[B' history-search-forward
So far it works. Thank you diath.