Generally Kafka takes advantage of the OS page cache and the fact that its data is sequential. The reason you can back Kafka effectively with spinning disks is because Kafka is just an immutable log.

OTOH you won't always have sequential access so services that store data in memory cover some of those other cases (eg a KV store like Redis).

I'm not familiar with Kafka specifically, but I reckon caching in memory is the way to work around that.

I think it's assuming that Kafka caches a lot more than your local machine, and therefore is less likely to need to access disk.

Note that it also assumes low network latency, though; if you have congestion within the data center, things change.