Hacker News Reader

Comment by londons_explore | Hacker News Reader

Is it me, or is "sub millisecond" actually quite unimpressive for an 'all-in-the-same-building-on-ssd' filesystem?

When my consumer flash ssd can typically provide data to a read request inside 50 microseconds, I really expect a big hosted filesystem to have a median latency in the 100 microsecond range.

Ie. I expect amazon to have the majority of filesystem metadata in RAM, such that a single read request from the user translates to a single SSD read request at a known block address. That data is then sent back across the 10GBit ethernet network to the machine doing the request, adding 1.3 microseconds per network switch.

View on HN

Do a ping test between two machines in your local network. Turns out it’s very hard to get sub millisecond speeds. Especially sub 500 microseconds.

the kernel will buffer and queue packets vs sending them out asap. So there’s delay added at your host level and networking layer. The. You’ve got the speed of light traversing that distance. All this adds latency.

But something like a filesystem should probably be built on RDMA to get the kernel mostly out-of-the-way. Then you should see performance like this:

https://www.microway.com/knowledge-center-articles/performan...

And the speed of light is 1000 feet per microsecond, so even the largest data centers won't add more than a handful of microseconds.

I just put a crossover cable between a Mac mini and a random Linux box and with no tuning the median ping is 180 microseconds.

Now add some routing and switching...

Or is there no routing between the storage and the application servers in a data center?

1.3 microseconds per switch is typical on 10GigE ethernet. I doubt your message goes through more than 10 switches even in amazons data center.

RTTs are extremely low in EC2, and I doubt EC2 is wasting any fibers on a mere 10ge link.

> Turns out it’s very hard to get sub millisecond speeds. Especially sub 500 microseconds.

I have to contest this a bit. The company I work for doesn't put much effort into latency from what I can tell, and sees ~200us between servers in the same data center, and ~400us to another data center 12 miles away.

Uh… no. PCI-E bus is about 900ns on each end, switch adds 300ns. You can get under 3us ping pong, you’re off by orders of magnitude.

Your sense is probably coming from bad OS stacks. Kernel bypass like Amazon uses achieves this easily.

It's not all in the same building though

Reply on news.ycombinator.com