A “64 bit clean” CPU would be nice, but practically you’d also want new 32 bit compatible CPUs for markets that need it. Gamers are still going to want to play old games with MORE POWER, business will want to throw more CPU at some process that is reliant on ancient code and so on. Apple has tried forcing the issue, and it didn’t exactly make everyone happy, and Apples view on compatibility is rather different to Microsoft’s or Linux / Linus’s.
So now you need to design and verify two CPU cores instead of one. The most efficient for an engineering staff and resource allocation perspective would be to just have the “x64-ng” core being the normal AMD64 core with legacy support lasered off. So probably not much in performance gains. If you had the designs actually diverge you’re going to end up with with duplicated work by more people / teams and thus less profit.
With the trend for dedicated low power cores, the companies already have two lines of core design to maintain, they aren’t going to want more.
I’m not saying it’s impossible, and a legacy free x86 core would be nice, but the business case for getting rid of 32 bit support probably isn’t there (yet?).
(You do also mentions Z series which has backwards comparability in some form going back to System 360 from the 60s - getting rid of this stuff is hard once it’s entrenched).