The M1's memory is LPDDR4X-4266 or LPDDR5-5500 (depending on the model, I guess?) which is about double the frequency of the memory in the Intel Macs.
Apparently, this alone seems to account for a lot of the M1's perf wins — see e.g. the explanation under "Geekbench, Single-Core" here: https://www.cpu-monkey.com/en/cpu-apple_m1-1804
Bleeding-edge-clocked DRAM is a lot more costly per GB to produce than middle-of-the-pack-fast DRAM. (Which is weird, given that process shrinks should make things cheaper; but there's a DRAM cartel, so maybe they've been lazy about process shrinks.)