LTO basically just calls back into the compiler and is multi-threaded on lld and mold at least, and probably anything else that supports LTO. And make sure you're using ThinLTO, it's still expensive but much faster than full LTO.
Have you tried the mold linker? I've been using it and it works pretty well, quite fast.