Hacker News Reader

For more comparison, here's a $5 Digital Ocean droplet (VPS)

      ==============================LINUX DATA BELOW===============================
      CPU                 : GenuineIntel Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz 2400MHz
      L2 Cache            : 15360 KB
      OS                  : Linux 3.13.0-37-generic
      C compiler          : gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.1) 
      libc                : libc-2.19.so
      MEMORY INDEX        : 35.003
      INTEGER INDEX       : 33.843
      FLOATING-POINT INDEX: 45.870
      Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38

How "reliable" are those numbers in estimating real application performance?

I've run nbench for a 1 CPU VM in EC2, and a 2 CPU VM in DO, and the former is a lot faster ( almost 2x ) than the latter!

    EC2 Ubuntu ( 1 CPU )
    ==============================LINUX DATA BELOW===============================
    CPU                 : GenuineIntel Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz 2400MHz
    L2 Cache            : 30720 KB
    OS                  : Linux 3.13.0-74-generic
    C compiler          : gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.1)
    libc                : libc-2.19.so
    MEMORY INDEX        : 39.370
    INTEGER INDEX       : 35.426
    FLOATING-POINT INDEX: 53.665

    DO CoreOS ( 2 CPU )
    ==============================LINUX DATA BELOW===============================
    CPU                 : Dual GenuineIntel Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz 2000MHz
    L2 Cache            : 15360 KB
    OS                  : Linux 4.2.2-coreos-r2
    C compiler          : gcc version 4.9.2 (Debian 4.9.2-10)
    libc                : libc-2.19.so
    MEMORY INDEX        : 20.668
    INTEGER INDEX       : 19.277
    FLOATING-POINT INDEX: 28.370

Edit: Here's also a 2 CPU machine on Azure

    ==============================LINUX DATA BELOW===============================
    CPU                 : Dual GenuineIntel Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz 2397MHz
    L2 Cache            : 30720 KB
    OS                  : Linux 4.2.2-coreos-r2
    C compiler          : gcc version 4.9.2 (Debian 4.9.2-10)
    libc                : libc-2.19.so
    MEMORY INDEX        : 28.667
    INTEGER INDEX       : 24.351
    FLOATING-POINT INDEX: 39.950

This is helpful but you're not mentioning what instance type at EC2.. t2 and t1 instances do heavy throttling after you use up your CPU credits[1] so it's likely that you were just using all or most of a full Xeon CPU. I got similar results on even a t2.nano.. until I used up the CPU credits. Then it slowed to a crawl!

Also keep in mind that nbench only tests a single CPU[2].

Apples to apples would probably be a small m3 or m4 against a comparable DO instance. (but let's not go down that cost disparity rabbit hole.. esp EBS vs SSD or bandwidth!)

1. https://aws.amazon.com/ec2/instance-types/t2/

2. https://en.wikipedia.org/wiki/NBench#Shortcomings

If nbench only uses 1 core, yes, my comparison is wrong. Is there a similar benchmarking linux tool that can use many cores?

edit: instance is a t2.micro

I don't think the comparison is wrong per se; this actually makes it a bit easier, since you're just comparing one core to another.

The only thing that I think is problematic is comparing a t2 to anything else, since that CPU performance is not sustainable as it might be elsewhere. m3 and m4 (and other instance types @ aws) are not explicitly throttled. (Source: AWS Solutions Architect)

We're running a portion of our workload at Userify[1] on AWS, but not the biggest portion, but this is actually for bandwidth cost reasons, not CPU (even though our workload is almost entirely CPU and bandwidth -- almost zero disk; our infrastructure at AWS would cost 8x more!)

1. https://userify.com (SSH key management)

I assumed nbench uses all the available cores, that's why I said I'm wrong. On a 1 core basis, yes, it's useful, and I get that this t2 instance is not a stable baseline. I only have that available right now, as my free credit.

> Is there a similar benchmarking linux tool that can use many cores?

There is kcbench:

http://linux.die.net/man/1/kcbench

It compiles a kernel.

Here are the results on a C2S:

  CPU                 : 4 CPU GenuineIntel Intel(R) Atom(TM) CPU  C2550  @ 2.40GHz 2394MHz
  L2 Cache            : 1024 KB
  OS                  : Linux 4.4.4-std-3
  C compiler          : gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.1) 
  libc                : libc-2.19.so
  MEMORY INDEX        : 22.134
  INTEGER INDEX       : 17.899
  FLOATING-POINT INDEX: 21.522

I have to be honest, they don't look bad at all.

From my experience so far, C1 is like working with a microwave, and VC1 (VPS) is much nicer, for €2.99 let's not forget.

And for comparison here's the VC1 (VPS):

  CPU                 : Dual GenuineIntel Intel(R) Atom(TM) CPU  C2750  @ 2.40GHz 2394MHz
  L2 Cache            : 4096 KB
  OS                  : Linux 4.4.4-std-3
  C compiler          : gcc version 5.3.1 20160224 (Debian 5.3.1-10) 
  libc                : libc-2.21.so
  MEMORY INDEX        : 22.474
  INTEGER INDEX       : 17.149
  FLOATING-POINT INDEX: 22.035

The single core performance is pretty much the same as the C2S, but it has half as many cores and one fourth the memory.

Here are the results for a Kimsufi KS-1 server which has an Intel Atom N-2800 CPU (1.86Ghz, Dual Core, 4 Threads) and 2GB RAM, 500GB HDD for 6€/month. I'm posting them here because they are competing in the low-end dedicated server space:

nbench:

    CPU                 : 4 CPU GenuineIntel Intel(R) Atom(TM) CPU N2800   @ 1.86GHz 798MHz
    L2 Cache            : 512 KB
    OS                  : Linux 4.2.0-25-generic
    C compiler          : gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.1) 
    libc                : libc-2.19.so
    MEMORY INDEX        : 11.834
    INTEGER INDEX       : 11.540
    FLOATING-POINT INDEX: 8.730

openssl speed results at http://pastebin.com/2dfPPx2t

I am still paying £3 a month for one of those! Shame they don't still sell them at those prices.

C1 (ARM):

    C1 root $ sysbench --test=cpu --cpu-max-prime=20000 run
    sysbench 0.4.12:  multi-threaded system evaluation benchmark

    Running the test with following options:
    Number of threads: 1

    Doing CPU performance benchmark

    Threads started!
    Done.

    Maximum prime number checked in CPU test: 20000


    Test execution summary:
        total time:                          686.1683s
        total number of events:              10000
        total time taken by event execution: 686.1557
        per-request statistics:
             min:                                 68.59ms
             avg:                                 68.62ms
             max:                                 70.83ms
             approx.  95 percentile:              68.62ms

    Threads fairness:
        events (avg/stddev):           10000.0000/0.00
        execution time (avg/stddev):   686.1557/0.00

vs the VPS (VC1):

    VC1 root $ sysbench --test=cpu --cpu-max-prime=20000 run
    sysbench 0.4.12:  multi-threaded system evaluation benchmark

    Running the test with following options:
    Number of threads: 1

    Doing CPU performance benchmark

    Threads started!
    Done.

    Maximum prime number checked in CPU test: 20000


    Test execution summary:
        total time:                          45.9858s
        total number of events:              10000
        total time taken by event execution: 45.9822
        per-request statistics:
             min:                                  4.59ms
             avg:                                  4.60ms
             max:                                  4.98ms
             approx.  95 percentile:               4.61ms

    Threads fairness:
        events (avg/stddev):           10000.0000/0.00
        execution time (avg/stddev):   45.9822/0.00

Just for the lulz, here is nbench running on my Asus Chromebook Flip.

    ==============================LINUX DATA BELOW===============================
    CPU                 : 4 CPU ARMv7 Processor rev 1 (v7l)
    L2 Cache            : 
    OS                  : Linux 3.14.0
    C compiler          : gcc version 5.2.1 20151028 (Debian 5.2.1-23) 
    libc                : libc-2.21.so
    MEMORY INDEX        : 15.111
    INTEGER INDEX       : 14.374
    FLOATING-POINT INDEX: 18.785

Very cool!

Here is mine (desktop PC from 2011):

    ==============================LINUX DATA BELOW===============================
    CPU                 : 4 CPU GenuineIntel Intel(R) Core(TM) i3 CPU         540  @ 3.07GHz 3059MHz
    L2 Cache            : 4096 KB
    OS                  : Linux 4.3.0-1-amd64
    C compiler          : gcc version 5.3.1 20160224 (Debian 5.3.1-10) 
    libc                : 
    MEMORY INDEX        : 37.273
    INTEGER INDEX       : 25.801
    FLOATING-POINT INDEX: 47.882

More results for comparison: http://www.tux.org/~mayer/linux/results2.html

Looks like it's around the speed of a 1GHz Pentium III.

But what about the VC1 (VPS)?

Wow, that's really low. I have a low power server at home that must be 6 years old at this point with higher numbers:

    ==============================LINUX DATA BELOW===============================
    CPU                 : CentaurHauls VIA Nano processor U2500@1200MHz 1200MHz
    L2 Cache            : 1024 KB
    OS                  : Linux 4.1.12-gentoo
    C compiler          : x86_64-pc-linux-gnu-gcc
    libc                :
    MEMORY INDEX        : 7.263
    INTEGER INDEX       : 7.365
    FLOATING-POINT INDEX: 9.665
    Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38

Benchmark from Intel i7 desktop

  CPU                 : 8 CPU GenuineIntel Intel(R) Core(TM)   i7-4790 CPU @ 3.60GHz 3910MHz
  L2 Cache            : 8192 KB
  OS                  : Linux 3.16.0-55-generic
  C compiler          : gcc version 4.8.4 (Ubuntu 4.8.4-  2ubuntu1~14.04.1)
  libc                : libc-2.19.so
  MEMORY INDEX        : 58.235
  INTEGER INDEX       : 52.421
  FLOATING-POINT INDEX: 79.472
  Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38

Here's a Intel Core i5 2.7Ghz on a MacBook Pro (Retina, 13-inch, Early 2015) compiled with clang (had to edit the makefile a bit):

    CPU                 :
    L2 Cache            :
    OS                  : Darwin 15.3.0
    C compiler          : Apple LLVM version 7.0.2 (clang-700.1.81)
    libc                :
    MEMORY INDEX        : 49.009
    INTEGER INDEX       : 41.947
    FLOATING-POINT INDEX: 92.671
    Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38

Benchmark for 20 euro Kimsufi(16BG Ram/1tb disk):

CPU : 4 CPU GenuineIntel Intel(R) Core(TM) i5-3570S CPU @ 3.10GHz 3706MHz

L2 Cache : 6144 KB

OS : Linux 3.14.32-xxxx-grs-ipv6-64

C compiler : gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.1)

libc : libc-2.19.so

MEMORY INDEX : 51.103

INTEGER INDEX : 48.049

FLOATING-POINT INDEX: 68.238

Baseline (LINUX) : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38

How does nbench compare to vpsbench?