erichileman

Why not run something like 8 x L40's for $4,750 a month from a bare metal provider like latitude.sh? This seems far more cost efficient and flexible.

See the other comment above re: perf stat. Working on the event descriptors to see and confirm the l1/l2 cache hits/misses.

Unfortunately <not supported>

Performance counter stats for 'php56 index.php':

        394.588620      task-clock (msec)         #    0.983 CPUs utilized
               226      context-switches          #    0.573 K/sec
                 2      cpu-migrations            #    0.005 K/sec
            17,447      page-faults               #    0.044 M/sec
   <not supported>      cycles
   <not supported>      stalled-cycles-frontend
   <not supported>      stalled-cycles-backend
   <not supported>      instructions
   <not supported>      branches
   <not supported>      branch-misses
   <not supported>      L1-dcache-loads
   <not supported>      L1-dcache-load-misses
   <not supported>      LLC-loads
   <not supported>      LLC-load-misses

       0.401580145 seconds time elapsed

Working on finding the event descriptors...

Memory seems likely.

On the E5 php 5.6 in top we see sys at 15%.

On the E3 php 5.6 in top we see sys at 7%.

On the E5 php 7 in top we see sys at 13%.

We are exploring memory perf more now.

We have not compared on bare metal.

We built 5.6 with -O2 -march=broadwell -mno=avx (had to remove, probably pecl ext issue).

There was about a 15% performance gain. Nothing that would explain the large difference between E3 and E5.

>>gcc -march=native -Q --help=target

>>march=silvermont

gcc native thinks this e3 is silvermont, a low power SoC.

Correct. We are going to build using the correct -march flags: https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html

Found this and will try it: https://github.com/centminmod/centminmod/commit/755dd9e87eac...

We thought that as well. The E5 has 4 memory channels max bandwidth of 51.2 GB/s. The E3 has 2 memory channels max bandwidth of 34.1 GB/s.

But we see a dramatic difference in single core tests. Our virtual machines have 2 cores assigned and there's also a dramatic difference. I wouldn't think that 1-2 cores would saturate 2 memory channels nor 34.1 GB/s bandwidth. If we were testing all 8 cores on the E3 vs E5 8 core virtual machine, yeah maybe, but 1-2 cores?

The L3 cache is much larger on the E5 at 20MB Smartcache vs the E3 at 8MB Smartcache. That seems to be the more likely suspect but I don't know enough about how the cpu cache is used in relation to php to say for sure. Hopefully, someone else does :)

Ref: http://ark.intel.com/products/88176/Intel-Xeon-Processor-E3-... http://ark.intel.com/products/64590/Intel-Xeon-Processor-E5-...

The binaries are from remi repo. We have a template we provision from. I used the same template on each virtual machine.

PHP 5.4.45 (cli) (built: Sep 19 2016 15:31:07) PHP 5.5.38 (cli) (built: Nov 9 2016 17:32:11) PHP 5.6.28 (cli) (built: Nov 9 2016 07:04:38)

The binaries are the same on each virtual machine. Are there build optimizations for E3/V5 vs E5/V2 that could make such a difference?