I devised `cdb` for this: https://github.com/jafarlihi/dotfiles/blob/a5133f49d67732199...

You can do just `cdb 5` to navigate 5 directories back.