Having more memory to address means you need more circuits that direct your reads/writes to the right place. Travelling a longer distance / through more complicated routes means the latency for each request will be higher on average.