AFAIK the bulk of link time optimization happens in the linker. The parallelizable compiler invocations just store extra information in the object files.

The linker invocation is a single make job, which is the unit of make -j parallelism.