Disk Locality in Datacenter Computing Considered Irrelevant (2011) [pdf]

A number of the assumptions in the paper are questionable in hindsight.

Disk bandwidth consistently exceeds network bandwidth by a significant factor in common systems and this is likely to be the case for the foreseeable future. A platform with a good I/O scheduler can easily and demonstrably turn that into extra throughput on the same hardware for many use cases.

Their corroborating examples are platforms that happen to have poor I/O scheduling. In these cases, the effects of poor I/O schedules would be expected to dominate the effects of differences in bandwidth. You would expect the differences in available bandwidth to have no practical effect as a result. But there are many platforms that do implement good I/O schedulers where differences in bandwidth materially affect performance because the I/O schedulers can take advantage of that bandwidth.

While not entirely their fault, their assumptions about the cost of SSDs is incorrect. The difference in cost/GB is down to a relatively small integer factor and shrinking, well under a single order of magnitude. And these can deliver enormous local bandwidth in cheap systems. In many cloud clusters now, the cost of a large, local SSD JBOD (how you would want to use it) is less than the cost of the server, even with cheap servers. The cost of using SSD over spinning disk has become increasingly marginal for many applications.

In summary: platforms with good I/O schedulers see real benefits to disk locality, assuming the application is I/O intensive. Platforms with poor I/O schedulers not so much. Make the appropriate choice for your use case and platform.

Also relevant: http://research.microsoft.com/pubs/170248/fds-final.pdf

The disk locality speed-up is increasingly not due to bandwidth, but latency (directly related to physical distance), which still matters.

To provide some perspective - in 1 cycle of a 1GHz CPU light (or electric potential) travels about 30cm. So round-trip communication with a disk (or whatever) that is 5cm away vs one that is one 1m away (could be same rack) would take 20 times longer in each direction. Now consider communicating across a football-field-sized datacenter.

This is really great for people who build out datacenters, but a large portion of consumers of compute+data will lease that from providers like Amazon and Azure, which means they cannot enjoy the benefit of the Disk I/o ~= local network i/o equation. There's no guarantee that a parcel of EC2 nodes is in a single rack, or even in a single datacenter, and so network i/o is not comparable to disk i/o.

Am I mistaken?

It's not really irrelevant considering the paper outlines that it's important to keep data in the same rack.

Disk locality is not that important when you're fully async, as throughput tends to be similar, however, if your code is not fully async then you usually pay significant penalties in latency, and if multiple nodes are able to access the data then you pay a huge price in caching.

The paper makes poor conclusions about SSDs/Flash - which are increasingly becoming the performance tier in the data center

For disk, sure, but not for Flash / NVM