Servers for an Accelerated Future

Who cares? What matters is price performance. They need to deliver value. What's inside, I don't care.

At least in my experience, for highly optimized code (which I imagine their's is), above certain core densities you run out of memory bandwidth, PCIe lanes, etc before you run out of cores unless your workload is particularly compute intensive. Power consumption also needs to be modeled into the hardware cost, so higher TDP does matter. It is a cost optimization problem, and those 64-core parts may offer little marginal gains for the added cost if bottlenecks are elsewhere in the system.

My experience buying CPUs for data intensive servers has typically been that maximizing the performance-density-cost curve often recommends a mid-range number of cores at a lower-middle clock rate. These CPUs are inexpensive relative to their product families while still having enough horses to drive your memory, PCIe, etc to saturation with highly optimized code. Just enough resources, but no more.

hey, this is Rami ... I lead HW team at Cloudflare. Great Q! .. in our case, there was a sweet spot between the number of cores, the L3 cache per core, NUMA latencies, mem bw per core, cost & power. The 48c gave us the best req per second per $, and it likely was due to a combination of all these things.

We have two more blogs that will come out probably today, which will shed more light on why AMD worked better for us.

If you look at how the chiplets are organized, you technically have 4 cores sharing a bank of L3 (2 of these 4 core groups per chiplet). In the 48 core model, 1 core from each 4 core group are disabled, so you have 3 cores sharing the same quantity of L3. So you now have 25% more L3 cache per core. You also have 25% more per-core memory and PCIE bandwidth.

If your workload is cache or memory bandwidth sensitive you might recover some performance despite having 25% fewer cores. You can probably run fewer cores at a higher sustained clockspeed. This may reduce a 25% deficit to something more modest like 5-10%, at which point the 64 core parts are harder to justify.

Does anyone know, is the 48-core SKU just a lower-binned version of the 64-core SKU, i.e. where one or more cores on each chiplet have flaws, and so AMD decides to just therefore build parts entirely out of e.g. “7 out of 8 cores enabled” chiplets to create the lower SKUs? (I know the chiplet-oriented manufacturing process makes it cheaper than ever before to get chips right the first time, but that doesn’t mean that they don’t have flawed chiplets sitting around that they could intentionally make use of.)

If so, maybe AMD doesn’t have high-enough yield on their 64-core part (i.e. 8-core chiplet sub-part) to satisfy huge bulk orders for them, without also generating huge numbers of the 48-core-binned SKU (i.e. 6-core chiplets, really 6-out-of-8-enabled-core chiplets) in the process.

And I would suspect that their production process is such that they do have a real, explicit 6-core chiplet part as well, which can be mixed-and-matched within a single CPU with the flawed, re-binned 6-of-8-core chiplets, giving them a powerful hedge on their own logistics (in about the same way that SPAM has flexibility in their ratio of chicken to ham that lets them ride out turbulence in either market, making the end-product cheaper than either input), but requiring even further that people consume the SKUs containing “6”-core chiplets.

I would bet that AMD very much wants to sell large buyers the lower-core-count CPUs, since their yield guarantees that—at least for now—they have so very much more of them, and attempting to make more of the highest-end part ensures that they end up with even more of the less-than-highest-end chiplets laying around.

AMD probably ideally wants order-flow of CPUs in a ratio, e.g. “1x 7742 : 8x 7642”, and offers both better deals monetarily, and far faster delivery (/less contention on orders with other clients) when you take them up on it; or when you buy huge numbers of 7642s alone, such that you’re consuming the cast-off from bullheaded clients who wanted pure 7742s.

"I can't imagine price would be major factor for a company of Cloudflare's size, but it's definitely true that the 48-core part is much cheaper."

I have worked at companies with 5 employees struggling day to day to keep afloat, highly profitable companies with employee counts in the 6 figures and multiple billions in revenue and just about everything in between. I have never worked at one where cost was not a major factor.

In Cloudflare's case revenues for 2019 were $287 million and they had a net loss of $105.8 million. They are competing with a market leader, Akamai, whose revenues for 2019 were 10 times theirs and had a profit of a few hundred million dollars, so I don't suspect Cloudflare is the exception to the rule of cost being a major factor.

It's likely L3 cache to core ratio: both 64 and 48 core parts have the same 256MB L3.

Yeah, I suspect it might be a memory throughput-per-core issue or one of the oldest ones in the books: they got a better deal for 48 cores as not all chiplet cores need to be operational...

(Disclosure: I work at Cloudflare but wasn't involved in this decision) Spectre variant 1 still affects patched chips with any sort of speculation and the price of fixing via compiler is substantial. It's a multi vendor structural issue. The hardware industry is stuck with some very unpalatable decisions to fix this. In some cases there are tricks to protect particular arrays but it's very finicky especially if arrays aren't a power of two in length.

>This major shift.....

Intel is still selling as many Xeon as they can make. And AMD's EPYC 2 / Datacenter revenue hasn't move up much relatively speaking.

Not only is it not major by any means, I wouldn't even agree on the word shift.

And that is speaking as someone who really want AMD to do better.

I'd guess they'd need to provision massively more to offer compute. They could offer it for some major data centres but I don't believe offering compute at all data centres would be feasible.

hey, this is Rami .. I lead hw team in Cloudflare.

Great observation! these chassis are designed for 19" racks, not OCP spec. But, who does not love green thumb screws!

This is Rami from Cloudflare ... we actually use DAC Cables w/ SFP+ for all of our 25G nics.

The orange cables in this pic (https://blog-cloudflare-com-assets.storage.googleapis.com/20...) are probably Cat5/5e/6 and are for management.

It looks like they're using DAC: https://blog-cloudflare-com-assets.storage.googleapis.com/20...

No GPUs, yet.

Yes, the 3TB is 2/3 used for Caching, and 1/3 as a local storage for our SSL/DNS/Firewall/etc.

What cache tasks are possible on GPUs? Maybe computing a hash, but that doesn’t seem like a big problem.