I suspect most of the performance improvements came not from being "64-bit", but rather from the new instructions in aarch64 (and even in armv7) over armv6 (which is what the 32-bit Raspbian has to use to be backwards compatible with the first Raspberry). So I don't think it is entirely appropriate to frame the performance improvements due to it going from "32-bit" to "64-bit".

In fact, hypothetically assuming the same instruction set, 64-bit can cause performance degradation over 32-bit, particularly in memory-bound benchmarks, because the size of memory pointers and default int size will take up twice as much space, which puts stress on the limited cache space, which will generate additional off chip memory accesses.