I don't think I've even seen an asdf plugin that directly uses a version manager. They generally seem to either A) download prebuilt binaries or B) fallback to building from source. Either way the installs go straight to ~/.asdf/installs.

The only exception I can think of is any dependencies on libraries that may or may not be installed by the system package manager. Even then, those dependencies are often optional.