AMD EPYC 7000 CPUs: 32 cores, 64 threads, 8 memory channels, 128 PCIe lanes | Hacker News Reader

AMD EPYC 7000 CPUs: 32 cores, 64 threads, 8 memory channels, 128 PCIe lanes

wccftech.com

242 pointsmrb93 comments

The Zen core had a pretty good (but no that amazing) success in the consumer market, because although people admired its multi-threaded performance, they were merely lukewarm about its single-threaded perf which "just" almost matches Intel. But oh boy, the Zen core in the server market is going to make a killing. Servers are all about multi-threaded performance (hence why 80% of the server market is dual socket). And it looks like a single socket EPYC is beating a dual socket Xeon... ouch. Finally a good kick in Intel's resting bottom.

I've used a dual cpu AMD Bulldozer for years (and still do), it's been rock solid and the 32 threads really helped with certain workloads. At the time the equivalent from Intel would have been far more expensive.

What board supports had dual socket FX? Those chips have come down in price to a point where it's now cheap to buy second hand systems for the price of the chip when it was released.

It would make a nice home VM server.

It's an HP server, I don't know if they make those any more. I can look up the model number if you want, I'm sure I have the invoice somewhere.

If you wouldn't mind grabbing a model number that would be great.

Ok, give me a moment.

edit: this is the one:

http://h18000.www1.hp.com/products/quickspecs/archives_Divis...

It's a DL 385 G7.

Cpuinfo: (there's two of these but they're obviously the same)

processor : 31 vendor_id : AuthenticAMD cpu family : 21 model : 1 model name : AMD Opteron(TM) Processor 6274

Looks like they are going for around $190 used on eBay with the CPUs and RAM included (some with HDDs and some without, but personally I'd buy without and use my own SSDs). Not a bad deal for the horsepower!

Bare in mind that nowadays there are used Opteron processors being given away practically for free. In fact, right now I have 6 Opteron 6172 sitting on my desk which I've bought off ebay for around 40€.

No, definitely not a bad deal. It will also take a ton of RAM.

> The Zen core had a pretty good (but no that amazing) success in the consumer market, because although people admired its multi-threaded performance, they were merely lukewarm about its single-threaded perf which "just" almost matches Intel.

That's highly variable, I was actually pretty astounded by it's single core performance and completely blown away by its multicore performance.

I built a 1700 (not even a +) for work a few weeks ago and I keep running into things where I pause and think something crashed because it can't have finished that fast...

Yesterdays was 5Gb of mixed data in 18s (turns out the SSD is the bottleneck), if I wasn't busy with a new job I'd be trying that 5Gb of data out of a RAM disk just to see how fast pigz (love pigz by the way, multithreaded gzip) can go with 8 cores/16 threads.

Pigz do fly.

Certainly for server software that is licensed per CPU and not per core, this is a major plus point for AMD.

> The Zen core had a pretty good (but no that amazing) success in the consumer market

In the communities I look at, the only Intel processors still getting regularly recommended are the Pentium G4560 and the Intel i7-7700K. And that's from consumers to consumers.

Also, when I look at my meta-benchmark for games, I see Ryzen with really good results by now. That changed a bit, it improved now that games are getting optimized and ram support is getting better. Before the i5 was still more viable.

I don't have insight into the whole market, but from my small observer position Ryzen does look like a pretty huge success.

While the raw compute performance numbers of Zen may be better than Intel, don't underestimate other economic and business factors. AMD has always played 2nd fiddle to Intel and it is hard to shake that perception. I'd love to see a major cloud provider (AWS, Google, Azure) actively buying AMD chips and making them available for compute. However, I am still a bit skeptical this is ever going to happen. There is just too much risk for a cloud provider. Intel holds market and nearly all mindshare in mainstream cloud computing.

Disclosure: $AMD shareholder

I wouldn't be so sure. As Intel's customers, the cloud providers are very interested in not seeing Intel maintain the near-monopoly they have inched toward over the last 5 years. They don't want to pay $7K each for the highest performance Xeon if it could be $6K in a slightly more competitive market. So much so that Google and others joined the OpenPOWER Foundation. Whether that was just for signaling alone, or if they legitimately thought a POWER8 chip would come available that met their needs, I of course do not know.

I agree, but there is inherent risk to switching for a cloud provider, even though AMD might offer a cheaper and faster CPU.

Certainly a risk, but this is Google, Amazon, Microsoft, Facebook we are talking about here. The largest public companies in the world. Some of the most technologically advanced as well. If they cannot bring pressure to Intel, no one can.

Cloud providers aren't going to switch fully to AMD, but I wouldn't be surprised if they buy a mix like 70% Intel / 30% AMD.

There may be a situation, however, where the servers that are rented to customers are 100% Intel which would perpetuate the appearance of a monopoly even if it no longer exists.

I disagree. It is _vital_ for cloud providers to keep Intel's feet to the fire. Things got so bad a few years back, Google started porting its entire software stack to ppc64le, which is far less convenient as a platform than amd64. I think we will totally see EPYC deployment in data centers if AMDs benchmarks hold up. You may not even know they're there, but I can guarantee they'll see a significant traction.

> AMD has always played 2nd fiddle to Intel and it is hard to shake that perception.

I don't know where you got that perception. Perhaps you're a kid fresh out of highschool and are oblivious to AMD's history, but AMD's Athlon line outperformed Intel's offering of the time by a wide margin, namely their Pentium II, Pentium III, Pentium IV and Pentium D lines.

IIRC, Intel only started to become competitive with AMD with the introduction of their Core 2 line.

In fact I am not an oblivious "kid fresh out of highschool"... Also, I think you are confusing performance with business success and sales. AMD may have had better performing CPUs in the past, but that has not mattered thus far. $INTC market cap 165 billion. $AMD market cap 10 billion.

Then you are probably aware that it has not mattered because of Intels shady tactics. [0][1][2][3] etc.

[0] https://www.wired.com/2009/12/ftc-sues-intel-for-anti-compet...

[1] http://www.silicon.co.uk/workspace/dell-pays-65million-to-se...

[2] http://techreport.com/news/8547/does-intel-compiler-cripple-...

[3] http://www.agner.org/optimize/blog/read.php?i=49#49

> Also, I think you are confusing performance with business success and sales.

I see your point. I've misinterpreted your comment assuming it was focused on performance instead of market share.

While that's true, Intel had better sales even in those times due to their aggressive marketing and business practices. IIRC AMD peaked back then at shipping 40 % of all units.

> I'd love to see a major cloud provider (AWS, Google, Azure) actively buying AMD chips and making them available for compute.

AWS used to use AMD chips in some instance types; I believe the whole m1 line used to be dual-sourced. If you got an Opteron when you launched it, you killed it and tried again, as the Xeon type was a little faster :)

Well I know what will be in the next stack of servers my company buys (in 20RU chunks). It's all Linux+Docker for dev & test with some KVM. Right now we use 2xCPU 48 core Intel, 2x1G and 2X10G. 1RU form factor holds two of these servers. It's all about thread scale out for us. The more containers we can run per server = faster build and test throughput. Pretty cool AMD. Happy to have you back.

TLDR: Intel's advantage of being able to clock higher gets removed because of heat in these high density multi-core chips. 1p 32core $2k. 2p 32core $4k.

"..14% advantage of cores per rack that ship with their Naples platform compared to Intel’s. On Intel, a singular rack will consist of 4704 cores while AMD’s Zen based Naples Rack will ship with 5376 cores.

There’s also 14% advantage in VM (Virtual Machines) per socket. Memory bandwidth sees a 33% advantage as AMD has 8 channels while Intel’s Purley platform is configured for 6 channels per socket. Intel platform also supports 24 DIMMs while AMD can support up to 32 DIMMs." "release 20th of June."

I just posted this yesterday about Naples/Epyc already having a huge advantage over Xeons for certain server workloads due to support for hardware-assisted SHA calculations: https://neosmart.net/blog/2017/will-amds-ryzen-finally-bring... , already supported by the Linux kernel and many open source crypto libraries.

I honestly had no clue this reveal was right around the corner. These numbers really do give AMD a fighting chance here.

I doubt many server workloads have SHA calculations as a significant bottleneck. Doing SHA256 5x faster (11 cycles/byte to 2 cycles/byte [1]) isn't that groundbreaking.

[1]: https://bench.cr.yp.to/results-hash.html

Surely these processors introduce another new level of NUMA dynamics? Each group of 8 cores has its own memory controller, own PCIe root complex, and then there is a crosslink between each group of 8 cores.

Up until now you would (potentially) have to consider which socket you are on, and where your memory or IO devices (PCIe) are.

Now you have the same considerations within a socket, as well as between sockets?

Previous Opterons were also MCMs with NUMA within the socket, although their performance was poor enough that many people probably never noticed they existed. If Intel makes cluster-on-die mode mandatory then they'll also have NUMA within the socket.

Actually the opterons were pretty competitive for many years, but of course stalled with the opteron 63xx series. They still scaled better than intel, but often lost on single thread performance. But if you were looking at performance per node on parallel tasks they often won by a substantial amount on price/performance.

However since 2012 (the opteron 63xx announce) AMD dropped the ball. First they were non-competitive on the dual sockets, and then a year or so later they were non-competitive on quad sockets.

Not sure why it's taken them 5 years to bring out something newer, but surprisingly the ryzen/naples/epyc looks surprisingly competitive.

> although their performance was poor enough that many people probably never noticed they existed.

I've read HPC papers demonstrating Opteron processors outperformed Xeons in BLAS applications by a wide margin due to their greater throughput. I don't really know if that assertion ever had any basis on reality.

Yep.

Not really.

So the Epyc is basically 4 ryzens. So you get 4x the cores, 4x the memory channels, and 4x the pieces of silicon.

So think of a single socket ryzen as a quad socket motherboard. In either case you have clusters of cores/cache connected to memory controllers and hypertransport. For most workloads a NUMA aware kernel does a pretty good job of minimizing hitting pages on other controllers. But it's not a particularly big deal when you miss, typically about 10% (latency and bandwidth).

AMD makes all the I/O pins capable of hypertransport (or whatever they call it now) and PCI-e.

This isn't particularly new btw. MCM (multiple chips per package) go way back to the pentium pro if not before. Intel Xeons are all single chip, but have similar on chip architectures. The 4,6,8 core chips are pretty simple, but the larger core chips have a ring bus for one set of cores, and another ring bus for the second.

But generally for most workloads the NUMA issues related to the newer chips isn't a particularly large hurdle from getting good performance. What I am concerned about though is how good the Epyc floating point is, I fear they are bragging about integer performance and not FP because they are behind on FP.

For most server workloads this plays a relatively minor role.

It's more than that, I think the cores are still in clusters of 4 that talk over infinity fabric.

But there's no avoiding some kind of complex inter-core dynamics at this level. The Intel alternative is a bunch of ring busses that have different speeds to each core from any point. And this design makes every memory access go over the infinity fabric, so latency might be surprisingly even.

Haven't been following this closely, but has the random segfault problem [1] been addressed? I would imagine this a bigger problem for servers almost constantly maxing out all cores and threads compared to a desktop/laptop... Just imagine the horror when you write safe Rust code and get hit hard by heisenbugs in production...

[1]: https://community.amd.com/thread/215773

While the full root cause has not yet been found or resolved the limited issues have been pretty reliably resolved by disabling ASLR. Whatever the root cause it is likely the issue can be fixed through BIOS/microcode updates.

The number of people affected are low. My Ryzen machine has only ever run linux and compiles a lot and has never exhibited this behavior. Also, most new platforms have issues, even new server platforms. These will be worked through during substantial validation server OEMs will go through.

Lastly, look up the errata list for any Xeon CPUs. Intel releases microcode updates for them several times a year to fix bugs. Modern CPUs are complex and will pretty much always have bugs. Luckily some combination of BIOS or microcode updates will almost always resolve them.

I am a Ryzen user, actually built my Ryzen system on its release day. For your claims on Xeon's bugs, just wondering when was the last time Intel had to patch the microcode to fix bugs that can continuously crashing day to day workloads such as compiling some linux packages?

In case you don't fully understand the situation - when Ryzen was released, it doesn't work with many memory modules on the consumer market, as of today, for pretty high probability, you still don't get the top speed of RAM you paid for, it crashes on day to day compilation jobs, the AMD GPIO linux module maintained/contributed by AMD is too buggy to run on Gigabytes motherboards, oh, let's don't forget the FMA3 bug.

Downplaying the issues causing troubles for Ryzen users do not get Ryzen better.

How about the intel fdiv bug?

My understanding is that BIOS updates have improved the speed, stability, and clock speeds available.

This is far from unusual, read newegg for any new intel socket/chip and the reviews are full of dimms XYZ didn't work with motherboard ABC.

Every motherboard manufacturer posts a list of compatible DIMMS they test with, unfortunately they are rarely the dimms available to consumers to buy in quantity 2-4.

Thus the market opportunity for crucial that does their own testing and has a generous replacement policy.

> For your claims on Xeon's bugs, just wondering when was the last time Intel had to patch the microcode to fix bugs that can continuously crashing day to day workloads such as compiling some linux packages?

Intel had an even worse one in 2014: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762195

Glibc merged a patch to use Intel's shiny new TSX transactional memory extensions when available to do hardware lock elision, except TSX was completely busted, and any time a process used pthread_mutex_lock, there was a chance that it would immediately crash, or corrupt memory silently. (In practice, this would happen all the time.)

Their solution was to release microcode that just turned off TSX entirely.

TSX has been an experimental new feature --- disable it and all is good again, since there is no production-quality software targeting it. On the other hand, to me it seems that Ryzen segfault bug cannot be simply eliminated. Disabling ucode cache might help. Disabling ASLR "pretty reliably" (per grandparent) resolves it. But I would imagine "pretty reliable", sans official investigation report from AMD, not reliable enough for those operating server farms, which jeopardizes the main value proposition of EPYC. By no means am I bashing AMD's new tech though --- it's just CPUs are so critical that you want strong guarantees on "from this ucode patch onwards, you won't get hit by this particular problem, at least 99.9999% of the time".

It was addressed by better ECC usage on the micro-ops cache in the latest AGESA as far as I know.

So, this means a 2P system can pack 64C/128T, 256 PCIe lanes and 4 TB RAM?

What a monster. Pack this together with a couple Quadro GPU accelerators and you got some serious allround performance.

A couple? We're looking at 16 full-bandwidth Quadro GP100's with 256 PCIE lanes. That's pretty awesome, in the fullest sense of the word.

I'm interested to see what AMD does on the GPU Compute front in the near future with this beast of a platform as a foundation.

Hmm, you'll need some lanes for disk IO, networking and other peripherals. And of course a motherboard (and a PSU!) that can actually feed such a monster - alone 14 GPUs would consume 3.2 kW, that's more than a normal 230VAC power cord can supply. Add the 400W of two CPUs, 100-150W for RAM, 200W for fans, RAM and other peripherals and you're looking at something above 3.7 kW.

edit: as a commenter above noted, the 2P version has only 128 lanes, too - which still means 6-7 GPUs. Enough power to last a few years, I'd say.

Ah. Bit of a shame...But yeah 6-7 full compute cards is still incredibly powerful - especially when paired with so many CPU cores.

Both 1P and 2P have 128 PCIe lanes, as in 2P configuration half the lanes of each chip are used for cross socket communication.

Ah, I had assumed a separate form of interconnection. Thanks for the info!

I think a 2P system still only gets 128 lanes. But still.

For the more boring among us (like me), swap the GPUs for 24-ish NVMe SSDs and a few 10G cards, and that is one hell of a DB server...

DBA here. I like where your head is at.

If the Zen 2 / Rome series brings down the power/heat a bit, that will probably be around the time I'm seriously considering another upgrade... my i7-4790 desktop has been really good to me for a couple years, but within another 2, may be looking around again. Though, the consumer variant will also be up for consideration.

Nice to see AMD competing again, knew they would get some ground in the server space looking at the Zen benchmarks on the consumer CPUs.

The bottom of the article mentions a supposed leak of a 2018 server chip with 48 cores. I wonder if that's just an MCM with 6 chips, made feasible by the smaller die size and reduced thermals?

Or maybe each die will be 12-core. It looks like AMD is trying to draw Intel into a core war.

Ah, yes, that's perhaps a more sensible way to go about it.

Kind of curious: How much lower you want them to go on the power/heat metric? Your current CPU is an 84W TDP part. A Ryzen 7 1700 has twice the number of cores and draws 65W with stock settings. The 1700 can get hot, but really only if you overclock it.

Of course more performance per watt is always better and I too hope Zen 2 can be even better. But right now, in certain scenarios, Ryzen is already leading in both performance per watt and per dollar.

Right now, the Ryzen 1800 isn't enough for me to jump ship, the 1700 is slower for most workloads, so sticking where I am for now. Though if the equivalent to an 1800x were closer to the 65W mark (Rome generation), I'd probably consider switching.

My workload is mixed, but definitely advantages to more cores/threads, I have several VMs in the background for development, docker, database, etc. And do some video re-encoding though that's mostly GPU bound.

I've had my 4790K for about 2.5-3 years now (iirc), and my last few upgrades have been around the 3 year mark... will probably push my current setup towards 4-5 years. I did upgrade to a GTX 1080 video card, well worth it, but don't see the point in much else, other than more cores, and getting past 32gb ram.

Keep in mind AMD and Intel measure TDP differently - AMD give average for a particular workflow / scenario, and Intel give absolute maximum, so you can't compare them apples-to-apples.

Intel's TDP is not the absolute maximum, but also a maximum-typical value, measured differently from AMDs methodology.

Remember the TDP for AMD is for the whole SoC. While with Intel you should include the PCH as well. At any rate you should compare the whole system with exact same components apart from the CPU. And the difference is minimal.

I understand that, but I have a few reasons for my concerns on TDP... The biggest is noise level, while my case now isn't absolutely quite, it's not bad, I don't want to have to have more noise, or resort to a full water cooling setup (I currently have a closed kit for my CPU, and stock on the GPU, with 4x 120mm case fans (including water block).

The other is the relative compute power... I'm pretty happy with what I have, could always be faster, but if my typical use only sees another 5% of performance, I'm unlikely to notice the difference. And I know that compute will expand some of the backend/unexpected delays on occasion, not sure the real world impact. So seeing similar TDP would be nice so I know noise won't be an issue combined with expanded cores, and similar single-core or slightly slower is okay.

A lot of this comes from farther back when I could upgrade every 18-24 months and see huge gains. It's been relatively stagnant the past decade now. I went from i7-860 to FX-8350 to i7-4790K that I have now, and frankly the changes have been mostly marginal. The extra ram and SSD has probably been more of an impact. 8gb, 16gb, 32gb respectively, and the first SSD was too small (many symlinks to HDD directories), and now running 3x 480gb ssds, and may switch to a 2tb nvme on the next upgrade cycle.

The GTX 1080 was offset by moving to a 4K display. So it's gains and balances... nothing slow by any means (I remember being able to make tea before my first XT with an MFM hdd would boot). But ever the more impatient with things. LOL, aside: at work, I have processes that I have to slow down, because the connected systems can't keep up.

It looks like this is putting 4 8 cores chips on the same package to scale up the cores counts, etc.

What are some of the technical limit to if AMD to 2, 4, 8x this approach?

Only power/heat? IO should not be hard since pins on MCM should be able to scale out easily, right?

Supposedly the big innovation of the Zen platform is its interconnect tech managing to scale performance nearly linearly with more chips. Or so AMD claims.

There's going to be a hit to I/O when one of those modules needs to access RAM that's on a different module's controller. The new bus seems to be really fast though, so I'm not sure how many you get have before it's a real problem.

I'd guess external IO (memory and PCIe) might be a problem. How are you going to route all those wires out from the socket?

Secondly, there's probably some economic argument as well. Too few customers willing to pay for a humongous MCM, and with the attendant wiring complexity requiring more layers for the motherboard, it might be cheaper to go to more sockets instead?

It's interesting that the EPYC 7601 comes at a significant premium to the EPYC 7551P - double the price for a 200MHz base clock increase and dual-socket support. Question for those more knowledgeable on data centres - is the modest performance gain and increased density worth it for the cost?

It depends on so many factors (hosting costs, design/availability of motherboards/servers, workload easily parallelizable across multiple servers, etc)

But Intel does the same. The 2-socket http://ark.intel.com/products/91768/Intel-Xeon-Processor-E5-... is superior in every single aspect to the 4-socket https://ark.intel.com/products/93796/Intel-Xeon-Processor-E5... but yet the 4-socket CPU is priced 1.6× higher...

> The 2-socket... is superior in every single aspect to the 4-socket...

Except density :) If both Intel and AMD do this, then I guess data centre admins do see the increase in density as worth the extra cost.

Does this increase in density have yet more additional cost implications in terms of cooling?

8S and 4S servers (or even bigger systems) have always been extremely expensive; 2S has been the norm for many years. Being able to stuff two 2S nodes in 1U pretty much negates the density advantage of a 4S system, without the quadratic increase in price that a scaled-up systems have. (Also, density is usually limited by power dissipation, so the lowered power efficiency of bigger systems becomes an additional con there.)

If you have limited physical space, I can see taking the premium in cost for things like storage or CPU power. Larger data centres also have the capital to spend to squeeze out the cost of those components.

Heh, except for the most important part of having twice the performance, twice the cores, and twice the ram inside a single node with a single kernel.

Not all tasks can be split across nodes. If you need 4TB in ram you have to get the more expensive quad socket CPU.

The differences between the 7601, 7551 and 7501 stuck out at me as interesting. The 7601 is at least $1300 more expensive for 200MHz more clock and 10W/25W (???) more TDP. Is that an artificial distinction, or do such high-TDP chips have a worse yield?

It comes down to binning of the same chip for AMD and there are people out there willing to pay the 100% extra for a 10% improvement.

You don't buy a CPU alone. You buy a whole server. If the CPU increases 1000$ in price to gain 10% performance but the complete server already is 20000$, it's quite worth it to go from 20000$ to 21000$ for 10% of performance.

Also if you factor in a performance critical software for 100000$ that runs on that single node, you will buy the fastest hardware that you can get for a few thousand dollars more.

It doesn't totally negate your point, but paying $20k for a server is far from the norm, and I imagine a dual-socket motherboard also comes at a significant premium.

This site is a rumor mill. So add some grain of salt.

Videocardz (the source) has been right far more often than not. Still a rumor, but credible.

I have been wondering how PCIe 4.0 and 5.0 will play out for AMD. Since changes of PCIe and DDR requires a changes of Socket.

For AMD there is no PCIe 4.0 chip in 2018. And PCie 5.0 is already out in 2019.

maybe they should simply skip 4.0

> Since changes of PCIe requires a changes of Socket.

> Since changes of DDR requires a changes of Socket.

Not necessarily, but usually done so.

Does anyone know if there is ARK equivalent for AMD? I would be interested in such a dataset for comparing against Intel.

Kind of, but not as good/detailed [0]

[0] http://products.amd.com/de-de

Reply on news.ycombinator.com