I made a plugin for that. It works well enough for my needs. https://github.com/danhper/fish-fastdir