As a proof of concept I wrote a Fish native version of the function that figures out what local version of a tool to use I do actually use it to show the currently selected version in my shell prompt. The asdf version was unusable in a shell prompt.
It goes through the parent directory structure up until the top level root directory and finds the first .tool-versions file and parses out the version of a specific tool.
The asdf native shim takes ~100 ms in Bash (no idea if it can be optimized, but I assume so). My Fish version takes ~1 ms.