At least with GCC they can sometimes merge the patches.
GCC on xtensa adds extra instructions in several places.
GCC on Arm Cortex-M0 has awful register allocation, and M0 has half as many registers as most ARMs...
The main thing I like about clang is it compiles byzantine c++ code much faster.
Some projects like llama.cpp it's like eating nails if you're not using clang.
So with projects like llamafile I usually end up using both compilers.
That's why my cosmocc toolchain comes with -mgcc and -mclang.
I do remember reading about LTO not working properly, you're either unable to link the kernel with LTO, or get a buggy binary which crashes at runtime. Doesn't look like much effort has been put into solving it, maybe it's just too large a task.