An EPYC NixOS Build Farm | Hacker News Reader

An EPYC NixOS Build Farm

grahamc.com

109 pointsnuriaion36 comments

This is quite surprising. I assumed they would get an EPYC server with higher core at similar price (due to the cost of the CPU) and get advantage in performance/dollar. I didn't expect EPYC to beat a similar physical core count intel server running at similar core count let alone by this much margin. Am I missing something here?

I'm saying this as someone who really wants AMD to succeed and push the ecosystem forward. I just want to make sure that I'm reading the results correctly here.

They are effectively getting 35% extra performance 60% of the price that's just beyond impressive.

EPYC is compared to a previous generation Xeon with lower TDP (105w vs 155/170w) and slower RAM (2400 vs 2666). The Xeon server is more expensive probably because it has 4x the amount of RAM.

Edit: actually, this is a dual Xeon machine so its TDP is slightly higher than EPYC's.

Doesn't that make it more impressive? The AMD build has a quarter of the RAM and still came out ahead!?

Unsurprisingly, the amount of RAM has little effect on a CPU-heavy workload.

For this workload, they could have had 16 GiB of RAM and get pretty much identical results.

> EPYC is compared to a previous generation Xeon with lower TDP ... Edit: actually, this is a dual Xeon machine so its TDP is slightly higher than EPYC's.

Comparing TDPs between manufacturers is invalid.

> Comparing TDPs between manufacturers is invalid.

Why? It's all Watts

"TDP" is not a figure resulting from a standardized process. Each manufacturer has a different (and sometimes changing) definition of it. E.g. for Intel it is a supposed power consumption given a certain load at base clock. Meanwhile with AMD it basically encodes the standard thermal parameters (Tamb, Tj(max), Rthjc).

«EPYC is compared to a previous generation Xeon»

That's not enough to explain. I highly doubt replacing the 12-core Broadwell Xeon CPUs with a current-generation 12-core Xeon (which would be Skylake) would catch up with the 30-40% performance lead that EPYC displays in their workload.

The Zen microarchitecture of the EPYC is quite impressive and it could just very well be a workload that Zen is particularly good at.

For example the fact EPYC can pack so many cores on a single socket means all the RAM and all its 64MB of last-level cache are "local" to every core (at worst accesses go through the Infinity Fabric.) By comparison, in Intel's case when one of the Xeon cores need to access RAM or cache on the other socket, accesses have to go through the QPI link between the 2 sockets, and it has higher latencies and worse throughput than AMD EPYC's Infinity Fabric.

The EPYC has twice as many memory busses.

False. The E5-2650 v4 has four memory channels, and with two sockets populated, you get eight memory channels, same as EPYC.

>They are effectively getting 35% extra performance 60% of the price that's just beyond impressive.

Lets just ignore the price point for now because the memory capacity are different.

But the performance, 24 Core AMD EPYC Zen is ~35% faster than 24 Core Intel Broadwell Xeon under the same frequency? Broadwell isn't even that old because Intel has only launched Skylake Xeon E5 not long ago. And the different between Skylake and Broadwell IPC aren't that much either. the max turbo speed are similar, and the Intel platform had the advantage of two CPU using two heatsink, higher TDP, higher heat dissipation more time for turbo.

And most testing has shown Zen to be about Sandybridge IPC performance. Something seems wrong here ( Or may be too good to be true )

I suspect that the Xeon processors are running at a lower frequency under load compared to the EPYC ones.

That's just a hunch though.

It isn't, because according to the spec sheet its all Core frequency are about the same as AMD.

Maybe their workloads were I/O bound and the EPYC server has faster SSDs?

I'm curious if anyone knows the test tool they're using here, in particular for lines like:

  $machine->waitForWindow("^Desktop ");

It looks incredibly useful.

Thanks!

Well there seems to be something called nix-shell, but I never heard of it before. It seams pretty useful. The link to plasma tests is provided in the article.

https://github.com/NixOS/nixpkgs/blob/49a6964a4250d98644da61...

Thanks!

This isn't nix-shell. nix-shell is a tool for spawning ephemeral software environments composed of nix packages, something a little like python's virtualenvs.

Digging a little bit, nixos' tests appear to be written in perl and are executed in a context that has access to a bunch of useful primitives for asserting these sorts of things. https://nixos.org/nixos/manual/#sec-writing-nixos-tests

The tooling used by NixOS is custom to NixOS. You can check out the test directory for many more examples.

I'll be writing more soon about how these tests work and how neat they are. They really help NixOS quickly release really high quality updates!

The tests are written in Perl and are invoked by Nix's build tools. You can skim the API at https://nixos.org/nixos/manual/#sec-nixos-tests

In this case, the test makes use of both xdotool and Nix's own testing library.

It's our custom tool. NixOS has testing infrastructure built on top of QEMU, and you can use Perl scripts like this in order to automate aspects of testing inside that environment by talking to the QEMU monitor -- for example, you can not only wait for a window, but click a button in that window, then also wait for a service to start afterwords, for example. (If you were testing GUI functionality).

Tests are written in a mix of Nix and Perl automation scripts like this; the Perl scripts control QEMU, but the Nix code describes what is put into the disk images. It's fairly bespoke and tuned to our own needs. The source code for the Perl driver is here: https://github.com/NixOS/nixpkgs/blob/f148c5c4a1ffa353836bfd...

It can even do multi-machine test setups as another comment pointed out, and it's pretty fun for writing functional demonstrations; for example I've been meaning to use these tools to write examples of doing PostgreSQL replication and failover, etc.

Thanks very much for this information (you and @grhmc).

PLEASE consider releasing some form of this tool, or at least documenting the snot out of it... it looks incredible. We'd really use this at work.

I think it would be really difficult to get such a good experience with something that isn't NixOS. You might look at things like:

- http://open.qa/ - TestKitchen

But I must recommend, indeed, taking a good hard took at NixOS! :)

The internals of the test framework are pretty heavily tied to Nix and NixOS, so a release independent of Nix(OS) isn't probably going to happen. With that said, the test infrastructure itself is exposed as library functions that any Nix user can use. And even though the test VMs itself will run NixOS, the host machine can be any Linux distro with Nix installed.

For anyone confused by instance types, this is for Packet.net

So these two servers are pretty similar, I originally thought they were comparing a Amazon m1.xlarge vs EPYC...

This is comparing far more than Xeon vs. EPYC, not even a single mention of the storage subsystem on these respective machines. It's not like the tasks being timed are 100% CPU-bound...

On the other hand, the EPYC server is 50% less per month so I believe it is showing how much more you can get for less $.

The EPYC server only has a quarter of the RAM though, so it’s really apples vs. oranges to compare prices directly.

I wonder if there were NUMA effects at play since the Intel server was dual-socket and the EPYC was not. It's not entirely surprising that a CPU-heavy workload reading from filesystem cache (i.e. memory) could cause a substantial amount of nonlocal reads that the single-socket server wouldn't have to deal with.

That said, that's just the first thing that popped into my head to explain the result. Either way, the results here are extremely encouraging!

A "single" EPYC is eight 4-core processors so there are even more NUMA effects there.

I think it’s the other way around: four 8-core processors

It's four dice with two groups of cores of four each connected by a callosum.

In NUMA-speak, each group is (probably) a node with neighbouring nodes having a lower distance.

We're now well out of my knowledge base, but your comment certainly makes things interesting. If they have similar NUMA latencies (big if!), I guess maybe the edge would go to whichever chip had higher NUMA bandwidth? Just sort of thinking out loud...

Or perhaps it has absolutely nothing to do with memory speeds, but I can't think of what else would have such a dramatic effect. I'd love to see the same test with a single Xeon.

Unless your buying top of the line Intel parts AMD is going to have more performance at most price points. It's the only logical decision unless you use AVX-512 or your workload is single threaded.

They are testing performance of a heavily modified and secure Xen type 1 hypervisor against a native computer with unspecified security policies?

This seems like a meaningless measurement.

Reply on news.ycombinator.com