Benchmarking High Performance I/O with SSD for Cassandra on AWS | Hacker News Reader

Benchmarking High Performance I/O with SSD for Cassandra on AWS

techblog.netflix.com

79 pointsriffraff13 comments

> Using sixty threads to write 2GB files at once using 64KB writes, results in 1099MBytes/s at 0.06ms service time.

It's not a good idea to scale disk writes using threads when you can use aio. You've introduced unneeded scheduling and locking penalty from threads. Additionally, such threads will be accessing and modifying kernel structures frequently and that's going to cause additional locking inside the kernel. They'll all be modifying same resources and structures while writing.

Using async write I/O wouldn't allow them to test the capabilities of the underlying hardware as writes would return immediately.

Also, sixty threads on such a powerful machine is not really a big deal. Even though 0.06ms service time seems small, it is large enough such that you won't have more than a handful of threads competing for CPU resources (they'd waiting for some I/O completion)

You certainly get notified when AIO O_DIRECT writes complete, so it is possible to benchmark AIO. For example, using fio.

> It's not a good idea to scale disk writes using threads when you can use aio

Last I checked (at least a couple of years ago admittedly) glibcs POSIX aio_* implementation was implemented using threads despite kernel support for it.

But there is a (nearly undocumented) native Linux AIO API.

And that's better?

It is better in the sense that it is actual AIO. Whether you want AIO so badly that you're willing to suffer the lack of docs is a case-by-case decision.

Yes. The io_submit based async io API is much better, especially if you are doing lots of iops.

The point you should take away from this benchmark is that a 3 year old Centos 5 kernel with no tuning, no filesystem tuning and default mdadm options did 100K IOPS or 1GByte/sec pretty much regardless of the benchmark options in use. It just needed enough concurrency to load up the disk. There are many other options on iozone for using different variants of posix IO and threading. I showed the basic options and the mmap option. So with careful tuning you could do IOPS more efficiently, but what I found was very accessible out-of-the-box performance.

for example the mmap read test used /opt/iozone/bin/iozone -w -s 2G -r 4k -i 2 -B -t 60 -F test0 test1 test2 test3 test4 test5 test6 test7 test8 test9 test10 test11 test12 test13 test14 test15 test16 test17 test18 test19 test20 test21 test22 test23 test24 test25 test26 test27 test28 test29 test30 test31 test32 test33 test34 test35 test36 test37 test38 test39 test40 test41 test42 test43 test44 test45 test46 test47 test48 test49 test50 test51 test52 test53 test54 test55 test56 test57 test58 test59

    TL;DR

    What follows is a more detailed explanation of the benchmark configuration and results. TL;DR is short for "too long; don't read". If you get all the way to the end and understand it, you get a prize...

I believe it's "Too Long; Didn't Read"

Ya, they've got it backwards. I was a little confused when the TL;DR was longer than the first part.

I sure they meant "too long; don't read" to fall in line with their "prize" of coming to interview if you could make sense of the explanations and benchmarks.

Reply on news.ycombinator.com