I used to agree with you until I saw this presentation by Jim Keller, one of the biggest names in chip architecture. It really illustrates how many more options we have to improve performance.
https://www.youtube.com/live/oIG9ztQw2Gc?si=vCBGGm0tkhG-VOyd