System design implications of declining network latency vs. disk latency

This has been a huge factor for our firm.

Trading systems used to be, by necessity, monolithic in design. Your risk system, feed handlers, portfolio management, strategies, execution management, and drop copy/logging systems were all built into one process due to poor speed of interproceess communication, even on the same vlan. You just couldn't get the required performance and certainty with a distributed system. I mean its nice that your distributed database is eventually consistant over the course of 500 milliseconds, but that's a couple of order of magnitude too slow

Now, networks are sooooooo fast and bandwidth is so plentiful that we can afford to split many of the major subsystems onto different machines.

It's important to note that we don't split these subsystems up for redundancy, I mean we aren't Netflix, we can't pull out major subsystems and keep trading. if one goes down we intentionally take down the entire system. You can laugh at me here if you'd like....

Rather we split the systems up so that we can test and upgrade them easier. Before the split, each upgrade was a major change that might have happened 2x a month, now we can upgrade individual systems 2-3 times a week. Agian, you may laugh now, but I put a very big premium on stability and certainty.

I remember talking to someone at Google who told me that their low latency RPC framework was considered to be one of their key technological advantages. Having made the move over the course of a couple of years, I'd be inclined to agree, a robust and fast RPC framework is an incredibly freeing piece of technology.

"Youtube, Netflix, and a lot of other services put a very large number of boxes close to consumers to provide high-bandwidth low-latency connections. A side effect of this is that any company that owns one of these services has the capability of providing consumers with infinite disk that’s only slightly slower than normal disk.[...]The most common counter argument to disaggregated disk, both inside and outside of the datacenter, is bandwidth costs. But bandwidth costs have been declining exponentially for decades and continue to do so."

I have, at best, 1 Mbit upstream at my house. The author states:

"A typical consumer 3TB HD has an average throughput of 155MB/s, making the time to read the entire drive 3e12 / 155e6 seconds = 1.9e4 seconds = 5 hours and 22 minutes."

but my upstream connection is three orders slower (in spite of costing the same amount). How am I supposed to get my data to the provider? Just emailing an attached photo is already painful...

These are really intersting ideas for business and the data center, but I think the author is out of touch with how bad consumer internet service is for us regular chumps.

For example, cloud providers pay less for disk than you. They may even charge you less than it would cost for you to buy it yourself. But how much storage do you actually use? For most companies your entire storage budget is probably a rounding error in your cost of doing business. Amdahl's law says if you're looking to save money there you're wasting your time, particularly if it comes at the cost of anything important.

Which it kind of does. Even putting aside the entire privacy debate, cloud storage creates third party dependencies. Put the only copy of your data on cloud storage and now you're dependent not only on the cloud provider, but also their ISP and your ISP. If any of them has downtime, so do you. If the cloud provider loses all your data, they say "oops" and have bad PR for a while, you go out of business.

And as the article mentions, these numbers are idealized. It is possible to build a network that has a 2ms latency to cloud storage, but you have little control over that. I currently have a 28ms ping to youtube.com. And it's not as if you can measure the latency and then make assumptions as if it will never change. If your ISP gets into a peering dispute you may go from 5ms today to 50ms tomorrow. If your ISP over-provisions its network tomorrow you may start seeing three digit round trip latencies when the network exceeds capacity. And it's not as if you have much choice in ISPs if this happens; if you have any at all the alternative could be as bad or worse.

But probably the most important thing to understand is that the question is not even relevant to most people. With the amount of memory supported by even fairly old servers, your entire working set may fit in memory and make disk access latency quite irrelevant.

Some of those network links are labelled "PCIe" or "SATA", and some are labelled "Ethernet" or "DisplayPort", but they're all high-speed serial links. A PCIe lane is only 8Gbit, comparable to 10Gbit ethernet.

This means that your computer "backplane" design can be de-constrained; you don't have to draw a hard distinction between elements in the same case and those in the case of the machine next to you in the rack.

A rotating disk is roughly able to saturate a 1Gb ethernet link. Maybe we should just go full SAN and put Ethernet connectors on the drives.

That said, as network speeds surpass what were previously internal system interconnect speeds it does make new design configurations possible. That said, the lack of predictability of those inter-connects makes things a bit more complex. If you're running stuff in AWS for example, and you are disk bound. Its probably worth your while to benchmark using a multidisk system running ZFS as a "local" NAS point for your disk storage. You might find that you can run a dozen or more instances of that single storage box for less money than a dozen machines with chunks of disk.

I wonder how pronounced this effect will be in places where bandwidth costs are not following the same downward trend as in North America. e.g. Australia:

https://blog.cloudflare.com/the-relative-cost-of-bandwidth-a...

e.g. Will data centers be designed differently?

But when moving across administrative boundaries, especially to a more ad-hoc environment, serious problems crop up. Some assumptions the author is handwaving away:

1. That 2ms to Youtube is to a single specific Youtube cache. If you're about to get on a plane, are you supposed to inform Google so that they can move your files across the country?

2. "The network is reliable" - what do you do while you're on that plane? DSL is still our national infrastructure policy so in many places that is the best that is available, especially if you want a choice of providers to avoid anti-consumer behavior.

2a. Or wifi. What is the latency of modern best-of-class consumer gear? I'm getting 16ms with a simple TL-WR703N. Never mind mobile...

3. Access patterns being so long-tail points to the continuing utility of end-user SSDs as non-volatile cache. The devices that would be most likely to drop them (cost, space, power) are generally on the worst network links.

4. The datacenter containing the notion of what data is authoritative (otherwise you wouldn't need to hit it for every access as outlined). What is the need for this?

5. The overhead of implementing oblivious transfer (or the alternative of being surveilled and having decreased economic power and freedom).

When you think in terms of datacenters, it's tempting to think that the next advancement is end-user computers becoming part of that system, but that is a skewed perspective. The pendulum swings back and forth, and I think we're pretty close to Peak Butt. The next deluge will be user-centric software that utilizes the butt for its strengths, but respects privacy and tracks the authoritative copy on user owned machines.

If you change an application to use remote storage rather than local storage, you may not get severe performance degradation in some cases, but your application will experience more failure modes than it did before.

What if you are doing some data-intensive computations where the data access pattern is fixed and computation parameters change?

This particular application wouldn't care about latency, since you can linearize or otherwise arrange your data on disk so it is in the optimal position for your application. Instead, throughput would be the limiting factor.

Has anybody done a similar analysis on this basis?

Idea would be that I could setup two separate racks with UPSes, specify a safety factor, then have super-fast shared disk/block available. (DB clustering or whatever.)