Not sure we can say that a linker is the same as a compiler in this sense, but if so, maybe it is indeed worrisome.
[0] https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=READ...
The standard compiler license exception (this applies to LLVM to, e.g.) says that any such code that gets combined in with your application code doesn't count. Note that it's still a potential license violation to use that code elsewhere (say, using those routines in another compiler).
This isn't a concern for linkers because linkers don't really provide anything in the way of code, everything being provided by the compiler as a compiler or language support library. The largest code it might add to your program is probably the PLT stub code, at best a couple of instructions long.