Of course it matters. You probably haven't written low latency server daemons. A busy memcached might be serving 20k to 100k requests a second, it's not serving 1 user only. 100k requests all being translated to memory vs disk reads will have a huge performance impact.

Many caches including Memcached support batch gets as well as pipelining, so network latency won’t dominate.