Hacker News Reader

Comment by growt | Hacker News Reader

I made a habit out of measuring the disk performance of cloud servers, so here it is:

  ubuntu@instantcloud:~$ dd bs=1M count=1024 if=/dev/zero of=test conv=fdatasync                                                                         
  1024+0 records in                                                                                                                                      
  1024+0 records out                                                                                                                                     
  1073741824 bytes (1.1 GB) copied, 11.0101 s, 97.5 MB/s  

  ubuntu@instantcloud:~$ sudo hdparm -tT /dev/nbd0                                                                                                       
                                                                                                                                                       
  /dev/nbd0:                                                                                                                                             
   Timing cached reads:   2180 MB in  2.00 seconds = 1090.44 MB/sec                                                                                      
   Timing buffered disk reads: 268 MB in  3.02 seconds =  88.85 MB/sec

View on HN

How does that compare to other cloud servers?

You can read my comment history for exact numbers. But for SSD storage it's not very good. Better than amazon EC2 though (t2 instance).

You should have a look to the IOPS instead bandwidth

  root@instantcloud:~# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=16 --size=4G --readwrite=randwrite
  test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
  fio-2.1.11
  Starting 1 process
  Jobs: 1 (f=1): [w(1)] [100.0% done] [0KB/18714KB/0KB /s] [0/4678/0 iops] [eta 00m:00s]

  root@instantcloud:~# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=16 --size=4G --readwrite=randread
  test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
  fio-2.1.11
  Starting 1 process
  Jobs: 1 (f=1): [r(1)] [100.0% done] [22307KB/0KB/0KB /s] [5576/0/0 iops] [eta 00m:00s]

Didn't know about fio, thanks!

Tested on random instance, without any io intensive application running

VULTR 2 GB INSTANCE dd bs=1M count=1024 if=/dev/zero of=test conv=fdatasync

    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 8.11451 s, 132 MB/s
    
    
    hdparm -tT /dev/vda
    
    /dev/vda:
    Timing cached reads:   25106 MB in  2.00 seconds = 12568.51 MB/sec
    Timing buffered disk reads: 496 MB in  3.02 seconds = 164.16 MB/sec

DIGITALOCEAN 2GB INSTANCE

    hdparm -tT /dev/vda
    
    Timing cached reads:   8926 MB in  2.00 seconds = 4468.52 MB/sec
    Timing buffered disk reads: 542 MB in  3.00 seconds = 180.43 MB/sec


    dd bs=1M count=1024 if=/dev/zero of=test conv=fdatasync 
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 32.0005 s, 33.6 MB/s

Did the same test on a VM I have in DigitalOcean... ( Ubuntu, 512 MB droplet )

~# dd bs=1M count=1024 if=/dev/zero of=test conv=fdatasync 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 2.35872 s, 455 MB/s

Add four spaces at the start of a line to mark it as code (both to avoid lines being merged and fixed-width font).

(yes: it's just markdown here).

I'm not sure how good is the test as it depends on the IO levels of the host where the VM instance is running.

This is a 1GB Debian Wheezy instance with regular HDD in Memset [1]:

    dd bs=1M count=1024 if=/dev/zero of=test conv=fdatasync
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 17.7142 s, 60.6 MB/s

I checked the host and its load is average. DO's 33.6 MB/s sounds poor.

[1] disclaimer: Memset hosting is my employer, I built some of this; but this is an honest test (personal VM running several services).

EDIT: formatting

EDIT 2: same test on a VM with SSD disk:

    dd bs=1M count=1024 if=/dev/zero of=test conv=fdatasync
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 3.48953 s, 308 MB/s

Again, the host has average load.

FYI - dd is not an effective way of measuring disk performance.

I'd recommend using fio to do this using libaio and direct disk reads/writes, and IOPing for basic latency tests:

* Random 4k read test for flash storage:

  fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test \
  --filename=test --bs=4k --iodepth=128 --size=5G --numjobs=12 --norandommap \
  --readwrite=randread

* And writes:

  fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test \
  --filename=test --bs=4k --iodepth=128 --size=5G --numjobs=12 --norandommap \
  --readwrite=randwrite

---

Here's an example here is a test storage unit I'm logged into at work right now (NOTE: THIS IS NOT ON INSTANTCLOUD!):

  root@s1-san5:~ # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test \
  --filename=/dev/md200 --bs=4k --iodepth=128 --size=5G --numjobs=12 --norandommap \
  --readwrite=randread

test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128

...

  Run status group 0 (all jobs):
  READ: io=61440MB, aggrb=2339.7MB/s, minb=199652KB/s, maxb=200362KB/s, mint=26167msec, maxt=26260msec
  
  Disk stats (read/write):
  md100: ios=15636294/0, merge=0/0, ticks=0/0, in_queue=1581403800, util=100.00%, aggrios=7864320/0, aggrmerge=0/0, aggrticks=128880/0, aggrin_queue=131372, aggrutil=100.00%
  nvme0n1: ios=7576253/0, merge=0/0, ticks=123812/0, in_queue=125748, util=100.00%
  nvme1n1: ios=8152387/0, merge=0/0, ticks=133948/0, in_queue=136996, util=100.

While the test is running is you'll see the storage performance (Note the MB/s in the third [] and iops in the fourth []):

  Jobs: 12 (f=12): [r(12)] [12.3% done] [2537MB/0KB/0KB /s] [650K/0/0 iops] [eta 00m:50s]

And when it's completed:

* Throughput: aggrb=2339.7MB/s

* IOs (Read in this case): ios=15636294/0

And then with IOPing to test latency:

* Here's an example of really bad storage latency on my crappy old rotational RAID array at home:

  root@nas:/mnt/raid# ioping /dev/md0
  4 KiB from /dev/md0 (block device 7.28 TiB): request=1 time=27.2 ms
  4 KiB from /dev/md0 (block device 7.28 TiB): request=2 time=15.7 ms

* Here's an example of pretty good storage latency on my new storage at work:

  root@s1-san5:~  # ioping /dev/md200
  4 KiB from /dev/md200 (block device 1.09 TiB): request=1 time=136 us
  4 KiB from /dev/md200 (block device 1.09 TiB): request=2 time=124 us
  4 KiB from /dev/md200 (block device 1.09 TiB): request=3 time=112 us

* And one more on a VM with storage provisioned over iSCSI to a very slow rotational storage array that's quite busy:

  root@nagios:~ # ioping /dev/xvda
  4096 bytes from /dev/xvda (device 15.0 Gb): request=1 time=11.6 ms
  4096 bytes from /dev/xvda (device 15.0 Gb): request=2 time=0.2 ms
  4096 bytes from /dev/xvda (device 15.0 Gb): request=3 time=7.1 ms
  4096 bytes from /dev/xvda (device 15.0 Gb): request=4 time=1.2 ms

---

OK So let's try this on instantcloud.io / scaleway:

* IOP/s - Random 4k reads:

  bs: 12 (f=12): [r(12)] [0.5% done] [31050KB/0KB/0KB /s] [7762/0/0 iops] [eta 37m:32s]

* IOP/s - Random 4k writes:

  Jobs: 12 (f=12): [w(12)] [0.2% done] [0KB/7848KB/0KB /s] [0/1962/0 iops] [eta 02h:29m:10s]

* Latency:

   root@instantcloud:~# ioping /dev/nbd0
   4.0 KiB from /dev/nbd0 (device 46.6 GiB): request=1 time=1.4 ms
   4.0 KiB from /dev/nbd0 (device 46.6 GiB): request=2 time=1.4 ms
   4.0 KiB from /dev/nbd0 (device 46.6 GiB): request=3 time=1.4 ms
   4.0 KiB from /dev/nbd0 (device 46.6 GiB): request=4 time=2.8 ms

Conclusion:

Good performance for small arm devices but not even close to even a single entry-level consumer grade SATA SSD.

That's rather poor, considering my laptop with FDE does about 50% faster:

    ▶ dd bs=1M count=1024 if=/dev/zero of=test conv=fdatasync
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 7.32253 s, 147 MB/s

Reply on news.ycombinator.com