https://llvm.org/devmtg/2017-10/slides/Ueyama-lld.pdf
There is a breakdown in those slides discussing what parts of lld are single threaded and hard to parallelize so I suspect single thread performance plays a big role too. I generally observe one core pegged during linking.