But the shadow stack concept seems much dumber to me. Why write the address to the regular stack and the shadow stack and then compare? Why not only use the shadow stack and not put return addresses on the main stack at all.
If one were to design a modern ISA from scratch it would make sense though.
I thought you were asking about the design of the hardware: it's designed that way because compatibility means that the vast majority of people want something backwards compatible.
Very little is fully "compiled from scratch" when you consider for example that libc is in the set of things that must be recompiled.