If cache size was the answer then Intel would have already stuck more on their CPUs, and AMD's 32MB on Zen3 would be carrying far more weight against the M1 than it is.

Bigger caches mean higher latency caches, it's not strictly a matter of bigger is better.