Every extant A64 processor needs to deal with ARMv8 instructions, thumb instructions, thumb2 instructions, shift encoding, microcoded multiple load/store instructions... It's true that the subset of the ISA that makes up the overwhelming bulk of instructions actually executed is very simple. But frankly that's true of x86 as well.
> when an ARM CPU is in A64 mode
That's... not the way hardware works. Those transistors are still there, it's not like you can make them go away by switching "modes". They still are ready to switch every cycle, and in any case having transistors that "aren't needed" doesn't make anything faster, because they all execute in parallel (or at worst in an extra pipeline stage or two) anyway.
Either you have a simple architecture or you don't. A64 is a simple ISA. Real CPUs have complicated legacy architectures.