Why Cloudflare Chose AMD EPYC for Gen X Servers | Hacker News Reader

Why Cloudflare Chose AMD EPYC for Gen X Servers

blog.cloudflare.com

284 pointsjgrahamc117 comments

I really wish AMD would focus on its tooling. The Intel VTune Amplifier is a fantastic multi-platform profiler that can help understand how effectively your software is using the hardware (Pipelining, micro-ops efficiency, cache usage, etc.).

If AMD could come up with something similar, it'd make their offering a no-brainer.

> https://software.intel.com/en-us/vtune

I think (without too much knowledge in this field) that AMD has had to play catch-up for so long in silicon that now the catch-up is complete, it will take some time to recognise organisationally that more focus on non-silicon is overdue. Silicon lives in an ecosphere, but without competitive products there's little use in investing in that ecosphere. If you suddenly get there, you need to change your game quite drastically.

But wasn’t the same the case that last time they “disrupted” Intel with their amd64 and dual core CPUs? They caught up, but never really invested in the tooling. And then they lost market share again.

Why would this time be different?

Different leadership this time, maybe the new crew understands this better.

There was the whole ftc sued Intel for anti competitive behavior thing last time where they locked oems out of buying Intel and AMD. Hopefully without that things will be more diverse in the market going forward.

https://www.cnet.com/news/intel-to-pay-amd-1-25-billion-in-a... Lll https://www.ftc.gov/news-events/press-releases/2010/08/ftc-s...

I remember back in the day (circa 2000) I was all about AMD and all my custom-built machines had this emphasis on AMD chips due to the floating-point calculations be that much faster than Intel's at the time. Intel really wiped the floor with them over the next bunch of years... It will be interesting should AMD have their comeback now :) Oh how the tables turn!

I doubt tooling is the problem that is slowing down AMD adoption. Compiler or special applications writers care about that. These are not the ones who pay AMD for their platform. It's the big businesses - most care about stability, predictability, reliability hence they stay with Intel. Those that are interested mostly in bang/buck, like Cloudflare, choose to go with AMD, tooling or no tooling. You don't need tooling to check that AMD is far more efficient now, you just do some basic load tests and the result is easy to understand.

If you have crappy tooling, the software other people write -- in house or commercial software-- runs crappier on your hardware.

But it's irrelevant if it works faster than intel

It’s not irrelevant if Intel has tools to let you close that gap easily.

Tools can't close a gap of entire cores, much less twice or more of them.

It's a matter of principle.

Some people don't want cores that you can't gather insight into how you are spending/wasting their time.

I'm not a direct customer of Intel or AMD, but in general I would trade off clock speed/cores/cost for insight into what my software is doing.

Extreme example: MKL on AMD is 3x slower than on Intel. Meanwhile, if you lie to it and say you're an Intel part, MKL is 20% faster than Intel...

Unless you're doing low-level stuff no developer is going to be aware that this even exists, much less capable enough to use it. There are more 'programmers' than 'engineers'...

An extreme example of the tooling gap: MKL. Intel made MKL and used CPUID and instruction set optimizations in a non-conformant way that penalizes AMD; it's a fair bit faster than OpenBLAS on (to-date) dominant Intel hardware.

If you make CPUID lie to it that it's an Intel part, it'll use AVX2, and Threadripper parts will beat Xeon by a 20% margin. Otherwise it's 200%+ slower.

Meanwhile, MKL has been a fair bit faster than OpenBLAS, etc on Xeon ("pretend to be Intel" MKL and OpenBLAS are about the same speed on AMD), so all kinds of software is linked against Intel MKL.

When you have core libraries tuned for Intel, either because Intel makes them themselves or because the tooling for performance tuning on AMD hasn't been present, Intel has a software tuning advantage that helps hide AMD's current silicon performance advantage.

This just reminded me of some older article on Numpy performance on Intel vs AMD [1].

I’m not posting this to bash AMD. Just trying to make point that the software side can matter a lot and from end customer point-of-view this can make a huge difference.

[1] https://www.pugetsystems.com/labs/hpc/AMD-Ryzen-3900X-vs-Int...

I was just going to mention numpy, or.. libc.

Sure, if AMD hardware is much better than Intels, an unoptimized libc might run better there. That’s not the point.

The point is that a libc optimized for AMD could run much better than what it does today.

Improving the performance of low level libraries for Intel hardware is trivial because there are a lot of great tools that help you with that.

If I had to improve the performance of some library on AMD hardware, I would get some Intel hardware and tools, improve the performance for Intel, and then “hope” that this improves the performance for AMD as well.

I personally have not used it but AMD does have a cross platform profiler available, AMD uProf. Thought I would post in case it was a situation of not knowing it exists instead of having used it and found it severely lacking vs vTune.

Yes, there are lots of stories of individuals and companies choosing AMD over Intel. For me, what's more interesting would be any contrarian view listing the compelling reasons to pay the higher price for Intel instead of AMD. Off the top my head, some reasons would be:

- Intel AVX-512 instruction set

- Intel 4 and 8 sockets

What realistic workloads have a better performance/cost advantage on Intel vs AMD EPYC?

If you're a home user (gamer, programmer), is there any reason to buy Intel today? I'm about to buy another motherboard & CPU and considering AMD for the first time. In the past, I avoided it because AMD had an incompatibility (e.g. VMware) that kept me on Intel.

(E.g. previous example of AMD showstopping bug affecting VMware: https://communities.vmware.com/thread/456094)

> If you're a home user (gamer, programmer), is there any reason to buy Intel today?

There's definitely not much. I believe absolute peak single-core performance can still be slightly higher on Intel which is important for gaming, but maybe the last few AMD chips also killed that. And there are also some tools that are not well optimized or don't yet fully support AMD yet. For example:

https://github.com/mozilla/rr/issues/2034

This year's release of Zen 3 looks very promising. Hopefully RAM manufacturers wake up and we also get a bunch more ECC options that aren't so overpriced and slow.

> There's definitely not much. I believe absolute peak single-core performance can still be slightly higher on Intel

https://www.cpubenchmark.net/singleThread.html It's still weird to see amd anywhere near the top of this list, a few years ago the first few screenfulls were intel.

>>but maybe the last few AMD chips also killed that.

It's definitely quite close but PassMark just shows that it's a tie and it's not necessarily representative of game workloads. It all depends on how much overclocking headroom there is. The enthusiasts for this kind of thing will throw great cooling at these chips and get the best out of them. I wouldn't be surprised if the advantage AMD now has in fabrication meant it also won there though.

Clockspeed/IPC. Intel still is a bit quicker on average in games, about 7% compared to the equivalently priced Ryzen CPU. Some games, especially older ones, it can be as much as 20% for some outliers that are still quite popular.

Intel is also tending to win in lowly threaded desktop applications because it can boost higher for longer. AMD's single-core boost is for quite short periods and only on one core with a quicker drop off as more cores are utilised, but Intel will maintain its boost in well-cooled conditions for many minutes or forever with a bios setting (many motherboards come this way by default although it's technically outside of spec) and it will happily also run the single-core overclock with all cores, again with a bios setting (also out of spec, also often comes turned on by default). AMD's CPUs are pushed much harder out of the factory and only 1 core on the chip can achieve the stated highest clock speed so it just can't do the same thing.

Intel ends up better somewhat better in low threaded stuff, games and applications, due to similar IPC but higher clock speed. But AMD CPUs beat it soundly for anything that can utilise more threads as at the same price you get 50% more cores than you do with Intel at least. Below a certain clock speed boost and price Intel just isn't worth it at all, neither are its productivity chips. 9700k, 9900K, 9900kS the soon to be released equivalent 10000 chips, those are still worth it compared to the 3900k, 3800k for certain gaming. Below that however I wouldn't bother with Intel at all and I certainly wouldn't bother with Intel at 60hz, it only really plays at high refresh rates or very particular games.

Saying that just be aware that the Intel CPU will probably age worse as threading continues to increase, all be it a very slow process. It really is going to depend on if you play AAA games or whether you play something particular or indy games and if they have problems with performance and are also lowly threaded.

The only thing that matters to me is that Intel has been screwing us over monetarily for almost a decade. I wouldn't care if AMD was 30% slower( it's not more like 5%) I'm giving them my money cuz they're not screwing me over

You are several years out of date. Please take a look at benchmarks and real usage from the last few months. With Zen 2, AMD has won the IPC game. Their Threadripper 64 core CPUs boost to 4GHz. There is just no alternative to this kind of performance to be found anywhere in Intel products right now.

Speaking specifically for gaming, it’s interesting to see trends for machine specs by top streamers - the ones pulling in millions per year who usually pay big money for the best most gluttonous machines available. For the most part you see them running the intel 9900K/KS line HOWEVER many of them were built in late 2018 when both that chip and the 2080TI gpus came out.

It’ll be interesting to see what standard they set this summer when the new Nvidia GPU comes out. I expect everyone will upgrade then including CPUs if they deem a different CPU available.

Also a slight tangent, it’ll be interesting to see how the competitive landscape shakes out after the pricing hike in hardware due to logistics slow downs in Asia due to all the quarantines going on. Linus TEch Tips just did a great video on this, maybe when the dust settles AMD will be the undisputed champion.

The only advantage 9900K/KS have for gaming is that their default boost is very high (KS is 5GHz). Overclock an AMD 3900X or 3800X, and that advantage is moot at best and nonexistent at worst.

Also, don't forget that these "streamers" are basically blank canvas advertisement machines, "putting out" to whoever pays top money, be it Intel, which is very active in that scene, the various motherboard companies which want to advertise their Intel-geared products and cooling solutions, which sometimes are geared toward Intel CPUs. The last two are changing slowly. In the last year, the amount of B450/X570 chipset motherboards flooding the market was staggering. It was very clear where the wind is blowing in the gaming segments. Intel is still a very high-paying force, but you will start seeing "streamers" also take the AMD route over time.

Those that use streaming PCs are almost always using Ryzen for that purpose. They add streaming machines to allow them to run CPU encoding for h264 since it produces a better image than NVEnc and it is very parallelisable and just a lot faster on Ryzen at the same price as a result. Gaming wise they are chose and kept 9900K(S)s for a good performance reason but they are also using Ryzen for video rendering and streaming purposes very often as it tends to do the scrubbing and rendering out quicker as well, but that also depends on the software you use as Premiere is mostly GPU limited except on certain effects.

As far as I can see the Youtube/Streamer types are either poor and choosing Ryzen as making videos is their bottleneck and its better bang for the buck or rich and choosing the best CPU for each circumstance across a range of computers.

Programmers that game have a different trade-off. A lot of compilers aren't all that parallel and many other tools aren't either. It highly depends on the language and tooling and the program you are writing. Maybe you can get use out of 32/64 CPU with what you are using, but if you are programming a language that is single-threaded mostly (Ruby) then perhaps there isn't a lot of point on anything more than 2/4.

This is the wider issue with threading scaling generally and having two companies chasing differing things. Intel is going hard for more clock speed with minor increments at the moment while AMD is going wider at somewhat lower clock speed. IPC wise AMD has an advantage in many cases but it's by no means always the case. It is complex to choose the right thing if 20% matters to you, but if 20% doesn't then go AMD because at the same price you get a lot more cores.

Actually I am not, I talking specifically about the 3800 and 3900 verses the 9900KS. Intel is still on average faster but whether that is worth the loss in other areas really depends. The area in which Intel is faster is narrowing (high refresh rate, certain games that are lowly threaded) and the advantage is smaller. When we were talking the Zen1 CPUs things were worse than this, more like 15% average behind and up to 35%. As its narrowed AMD has also made plenty of wins too, it is now highly dependent on games played and if something you always play favours one over another.

Whether you then choose Intel depends on the circumstances, but the facts are the facts.

But you are. Facts are facts. Here are some facts:

https://www.cpubenchmark.net/singleThread.html

AMD CPUs are leading the chart with lower clock rates than Intel's 9900KS (4.0 GHz base, 5.0 GHz boost). Overclock those AMD CPUs and Intel has no advantage.

That isn't games, passmark is a wide splattering of applications, some of which clearly benefit from threading. I have talked about a particular set of circumstances in which Intel wins (and looses) and your source is garbage for that circumstance.

https://youtu.be/CGQY9yJqfMs?t=449 <- this is a 36 game roundup from a place that does decent game testing.

Here is another from GamersNexus probably one of the best places for gaming reviews out there https://www.youtube.com/watch?v=yqQ2X1y0jvw

Here is Anandtech on the 9900KS that includes a 3900X, their methods and range of tests is pretty outdated nowadays but still the same picture emerges https://www.anandtech.com/show/14979/the-intel-core-i9-9900k...

The majority of the big outlets say the same thing about gaming and low threaded workloads on these CPUs. Intel wins that still, not by as much as it did but it still wins most of the time and on average about 7%, at worst more like 20-25%. Intel also looses sometimes, it really depends.

The posted benchmark is single thread performance. So AMD is getting no multi-thread boost.

I'm not considering AMD because I'm afraid of stability issues. Too many things that's just unacceptable for me happened with AMD CPUs in the past (kernel panic when compiling stuff; RNG issues zeros after suspend and so on). I don't really need that many cores and price difference is not that big to consider moving from Intel. But that's for my home usage. Anyway I'm really happy that AMD exists and releases good products, because Intel slashes prices like crazy.

Another practical issue is running hackintosh in VM. It was not possible on AMD few years ago and even if it's possible now, it's unlikely as stable and I want to keep that ability.

> I'm not considering AMD because I'm afraid of stability issues

What about the plethora of security issues that are present in Intel but not AMD CPUs? Not sure there's a case to be made that Intel CPUs have less bugs.

Those are theoretical threats for personal computer. I don't care about them and I'm disabling software mitigations on my computer.

For home users, nothing. For larger businesses, it’s all about dollars, and there are some good use cases for AMD.

I would say that in the past, some OEMs have very fickle support for AMD lines, and you can run into problems if you have many of them affected by some obscure bug that isn’t getting fixed.

> If you're a home user (gamer, programmer), is there any reason to buy Intel today?

If you need EEC ram?

I know technically AMD supports it for their Ryzen line, but everything I’ve read says it’s up to the motherboard manufacturer to support it. And even if the motherboard manufacturer says you can use ECC ram, it doesn’t mean it will actually “use” it correctly.

So you mainly need to rely on others doing tests on their own systems to confirm if the motherboard actually uses it.

My understanding is that you need to use the EPYC line for guaranteed ECC usage, which is really expensive.

Intel’s W series Xeons start off at a reasonable price, and there are even cheaper Celerons that will use ECC ram.

> I know technically AMD supports it for their Ryzen line, but everything I’ve read says it’s up to the motherboard manufacturer to support it.

AFAIK every ASRock AM4 motherboard supports ECC.

Also, most Intel processors do not support ECC at all, except the ones which are competing (in some limited sense, since they are being absolutely crushed at most price points) with Threadripper, in which case, ECC is available across the board (and glorious quad-channel memory).

It is the same thing with Intel - motherboard must support ECC and actually use it.

Anecdotally, I've heard similar issues with AMD chip-sets and RAM.

While we haven't seen it for a while, this is real competition. In response Intel dramatically dropped prices (by over half) of their Xeon's within the past couple of days. And hopefully they'll refresh their line again.

https://fuse.wikichip.org/news/3352/intel-refreshes-2nd-gen-...

I think the real reason people are sticking with intel is the support and supplychain.

Intel have a reasonable two-way communications system with big customers, they also write their own firmware still, instead of slinging it out to contractors (although AMD are apparently changing that)

crucially they are able to supply 500k procs at a fixed price in a fixed deadline.

AMD are yet to be able to do that for big non-consumer suppliers(is what my hardware contacts tell me that is. )

For a home user I would think a relevant aspect might be the ready availability of well-integrated systems like the NUC. I just replaced a NUC with a new NUC even though I am aware of Zen2, I know it has its qualities, and I know how to build computers.

I'm interested to know where people use AVX-512. Off the top of my head, I can think of machine learning and crypto. Machine learning---that is, matrix/tensor stuff---seems more like a GPU workload. That leaves crypto, but most crypto (eg ChaCha or Keccak) only make use of 128 or 256-bit extensions (in fact, they seem to designed with this constraint in mind.)

Video encoding comes to mind.

Depends on your use case but the most recent GPU encoders are pretty solid actually. Used to be that they were awful for quality, but more recently they seem to have vastly improved.

The person above was asking what people use AVX512 for and video encoding is a big use case, better GPU encoders don't change that.

The person above also pointed out that things like machine learning are often better done on a GPU anyways. Certainly if you offload your video encoding to your GPU it does change the value proposition of that entirely.

I'm not sure what you are trying to say exactly but it seems like you are making some sort of prediction of the future.

Video encoding, especially newer codecs like HEVC and VP9 are CPU intensive and SIMD optimized tasks that people are doing right now.

I'm not referring to any sort of future prediction, those people can switch to GPU encoding of video right now and enjoy better performance.

https://developer.nvidia.com/video-encode-decode-gpu-support...

They asked what people are using it for right now and video encoding is one of those things. This is very direct. They didn't ask about solutions or what they should do about a particular problem, just about what people are currently doing.

There are diminishing returns when evaluating recursive formulas using SIMD, as the cost to gather/scatter into the vectors, duplication of computation, and transfer overhead to the GPU cost more than doing it in a single core with vector instructions.

Real-time audio processing is probably the most common case where you see huge gains in SIMD on the CPU but very minimal gains on the GPU. Other examples are certain polynomial approximations and physics simulations.

Also string manipulation, parsing, evaluating a regex. The overhead of getting to the GPU outweighs keeping it on the CPU core.

https://cpu.userbenchmark.com/Compare/Intel-Core-i7-9700K-vs...

Comparing the similar priced Intel and AMD CPU's shows Intel has a slight advantage.

Reading the conclusion, it seems for Overclocked performance, AMD is the winner, especially when factoring in large core workloads.

As games and typical "home-user" scenarios are not designed to take advantage of large multi-core CPU's, Intel would appear to be the better choice at the ~$350 CPU level.

I mean gamers probably shouldn't be buying a processor for $350 unless they plan on doing parallel workloads anyway. The tier below it won't be a bottleneck.

dont use this site. they do all kind of shady things to make Intel CPUs look better

https://www.reddit.com/r/buildapc/comments/chco8h/userbenchm...

Thank you. I was not aware of that.

The rr debugger, which is an amazing productivity enhancer for C/C++ development, still doesn't support AMD: https://github.com/mozilla/rr/issues/2034

i wonder why doesn't AMD sponsor these kind of projects to get maintainers to support their CPU. it helps both parties.

If you look at the rr issue, there is plenty of interest from the rr maintainers (of whom I am one), and also some engagement from AMD. The problem is there is a deep bug in AMD's performance counters or their kernel PMU driver, it's really hard to figure out, and the AMD staff best placed to figure it out are (I assume) really busy.

At FAAG its well known that some AMD procs are faster, cheaper and cooler(well higher performance per watt). None of that is in doubt, what they are having trouble with is supply.

It appears that AMD are not able to provide either the support (microcode tweaks, firmware heads up, firmware testing and bug reporting) or the volume to be useful.

The conclusion was that AMD want to nail consumer first.

SourceHut runs its performance critical infrastructure (git, builds) on AMD EPYC. These CPUs are sweeeet. The hard part is sourcing motherboards. I currently buy them from some dude in Germany on eBay because he's the only consistent source I've been able to find.

> SourceHut runs its performance critical infrastructure (git, builds) on AMD EPYC. These CPUs are sweeeet. The hard part is sourcing motherboards. I currently buy them from some dude in Germany on eBay because he's the only consistent source I've been able to find.

Heh, you know that guy is with / getting a visit from the German BDN now.

*BND, from Bundesnachrichtendienst

Why, is it illegal?

Two years ago they switched to ARM https://news.ycombinator.com/item?id=16646701

We tried to switch to ARM. We maintain our ARM port of all our code for the day there are servers we can use. However, we are in production with AMD now.

From one of the posts: "Readers of our blog might remember our excitement around ARM processors. We even ported the entirety of our software stack to run on ARM, just as it does with x86, and have been maintaining that ever since even though it calls for slightly more work for our software engineering teams. We did this leading up to the launch of Qualcomm’s Centriq server CPU, which eventually got shuttered. While none of the off-the-shelf ARM CPUs available this moment are interesting to us, we remain optimistic about high core count offerings launching in 2020 and beyond, and look forward to a day when our servers are a mix of x86 (Intel and AMD) and ARM."

In two years followed by "Why we are switching from Intel to RISC-V" and two years after that "Why we are switching from Intel to POWER".

No love for itanium? Aww.

I know this is sarcasm, but:

I personally never used Itanium, but the features it had at the time were very compelling. Unfortunately, it was too ambitious, and now it’s faster to do OOO execution instead of making the compiler do all the heavy lifting.

I don't think it every was a promising approach to let the compiler alone do that work. For one, it means that you basically have to recompile for every microarch, and the compiler needs all those details. But even then, the compiler can't fully predict cache behaviour etc. Which means ooo execution still is important.

I have never seen anyone have any love for Itanium.

It's such a beautiful architecture, but nobody seems to touch it and get out unharmed.

What about it is beautiful? There idea of VLIW?

Yes, the idea of optimising at compile time so you don't have to do OoO and get a simple but functional chip.

I think my stance on that is https://news.ycombinator.com/item?id=22459095

Do you see any ARM chips on the horizon this year or next that might change the equation? It looks like Amazon is seeing 40% better performance per watt (I think they said the power draw is comparable somewhere) with their custom design, will a commercially available chip be able to match that soon?

Side note: Thanks for making a great product and keeping it cheap/free for hobby projects! I got comfortable with the basic features on a small project and now advocate for it's use regularly.

We continue to look at ARM and test things. We continue to look at Intel and test things. Right now AMD makes the most sense for the servers we are deploying today. But don't be surprised if we have more Intel in our future or ARM.

Ampere QuickSilver is looking good.

Dear John,

As a happy Cloudflare user (free plan for personal use tho) and stock investor, I'm really pleased to see all your excellent and hard work and appreciate open sharing!

I've been tracking your ARM effort for a while (and did my math when you released Gen9 server! See https://news.ycombinator.com/item?id=18190891). I'm not sure how much you could share on what happened to Falkor and Qualcomm's ARM server business in general, but if possible, please do have a similar writeup. It would be extremely interesting and valuable to the whole industry.

Meanwhile I'd assume this must have been discussed internally, but have you ever considered designing your own ARM SoC (and boards) given that the licensing model actually could work (compared to the impossibility of licensing x86)? I know Cloudflare is still a small company (compared to the big four) and maybe the timing is not right and/or scale is not yet there, but you do have this option in the long run. If possible, I'd also like to hear your thoughts on this.

Anyway, thanks for the great writeup and keep the good work going! ^_^

We're not yet at a size where designing our own ARM-based SoC makes sense.

I see. Thanks for the straight answer!

> Notably, Intel is not inside.

I can hear the loud silence of that full stop.

Now i wonder if these were considered/known

https://www.servethehome.com/ampere-altra-80-arm-cores-for-c...

I’m a big fan of CloudFlare. I own a little stock, and I moved our company over to it after running on a competitor for about a year. CloudFlare has been much better. Whenever I use the front end, it just feels like it’s made by people who actually need to use this stuff, and it is so straightforward. The api is great too. When we were with the competition it was ridiculous how often we had to have an engineer assist us with setup because the front end didn’t support some feature we needed. With CF we signed up one day and have never had to have manual changes made by support, and the health checks, load balancing and failover have worked from day one. With the competitor the health check story did not live up to the reality and we had several outages related to the failover system not recovering properly.

When they publish these blog posts, it shows to me that they are willing to consider bold moves and that they aren’t afraid to be different. Also, that their architecture has been designed with this agility in mind. I’ve visited their offices and while their corporate culture didn’t seem like the best fit for me, I liked the team and what they are doing.

Come on, throw their competitor under the bus!

I'm guessing it was Akamai. Every change we need to make to our Akamai distros is an up sell conversation with them. It's obnoxious. I much prefer Amazon CloudFront.

Not Akamai but similar experience here. I've heard good things about CloudFront, but I believe it's mostly on the CDN side? Does it do things like authenticated origin pulls (client cert from provider presented to your origin), automatic SSL certs, health checks w/ failover, DDoS and WAF protection and unlimited bandwidth?

The WAF was one of the reasons we switched, our existing provider wanted to charge us 10x more just for the WAF than the entire Cloudflare plan is now.

Fastly?

I'm also rather interested to know if it is Fastly, I am considering switching away from them. The control fastly offer through the VCL is great but cloudflare workers look really great.

Resources on how to switch from VCL to Cloudflare Workers: https://developers.cloudflare.com/workers/archive/recipes/vc...

Akamai interface is like a jigsaw puzzle with several key pieces missing. Doesn't matter how much effort you put into it, how many times you did it before or how many documentation you read, part of it will still depend on evil incantation chanted to their support/billing people.

At least it was about 5 years ago when I last used it, I ran from that as fast as I could.

It's still like that and everything you might want to do is disabled by default. You have to go through a sales/contract process to get each feature turned on inidividually.

They should just set the price for each feature and charge us based on usage. We're actively migrating to CloudFront because we're sick of playing these games.

"it shows to me that they are willing to consider bold moves and that they aren’t afraid to be different"

Because they benchmarked a drop-in-place commodity replacement as being a better performance-per-dollar for their operation? I'm not sure what's bold about that. I find the notion that they posted a blog entry about this pretty bizarre -- almost rinky-dink. It seems like it's pandering to the AMD fanbase (which I am right now, just as anyone who is paying attention is, but I can still recognize it)

EDIT: Sorry, I didn't realize I stepped into a Cloudflare circlejerk. Apologies and carry on. So brave!

There's a ton, just a ton of intel-specific integrations and Cloudflare's switch to AMD was not just a "drop-in-place" move.

What "intel-specific integrations"? Cloudflare specifically talks about how they keep their stack runnable across a wide-range of platforms, including even entirely different instruction sets (e.g. ARM). Their own words support my statement.

I'm pretty sure it was a "drop-in place" move, and they say absolutely nothing to the contrary. They benchmarked a system that performed better than their existing system for whatever criteria they targeted, and chose it. Story at 11. Saying they "abandoned" Intel is uproarious given that if Intel pushes out a new 128-core Xeon at a decent price, they'd as easily "abandon" AMD for Intel.

As I say in another comment... AMD makes sense for us today because of the performance per watt. We continue to keep our software ARM-ready and look at ARM-based solutions. We also continue to look at Intel. We'll use what gives us the best requests per watt.

But it is notable that these are our first servers wit NO Intel components at all.

Don't mean to hijack, but I'm definitely curious.

Since you're the CTO of Cloudflare, and you posted the OP; Why did you replace the word 'Chose', in the linked title, with the word 'abandoned' here?

That comes across as real click-baity.

I'm curious why requests per watt is an important metric?

Are you running up against some sort of heat limit or optimizing for electricity cost?

Seems like requests per amortized dollar all in would make more sense. Or requests per cubic foot of data center space if that is your limitation.

This is a good question and I would like to learn the answer too.

According to some napkin calculations, the CPU is $5000 and will be kept in service, I guess, 5 years.

The US average price is 13 cents / kWh so this 225 watts CPU will cost $1280 in electricity over those 5 years.

Colocation price is probably low, let's say $50 / 1U server / month. So that's another $3000.

Why is electricity the most important here? I guess because you can't lower the other 2 costs very much? But isn't the CPU price itself based on the power savings it does provide?

Data center space is generally plentiful. Power and cooling, which for most purposes can be considered equivalent, are the scarce resource.

CloudFlare is trading half their server density[0] for a ~25% gain in performance-per-watt and performance-per-node[1]. Power is the whole ball-game at scale.

[0] https://blog.cloudflare.com/a-tour-inside-cloudflares-g9-ser...

[1] https://blog.cloudflare.com/an-epyc-trip-to-rome-amd-is-clou...

Ah, so it's not about power being a big expense, but being a constraint that is limiting their growth:

> We are constrained by the data centers’ power infrastructure which, along with our selected power distribution units, leads us to power cap for each server rack.

intel-specific? I thought AMD was just a drop in replacement for Intel since both same architecture and AMD was a clone of X86 originally, but I guess Intel licensed them the tech so they wouldn't be a monopoloy... but I know that's a bit of a history lesson.

Or did you confuse ARM and AMD? ARM I know you need to recompile your code but possibly other changes depending on what... and I believe there's even compatibility difference between ARM chip versions like 6 and 7, etc. but I guess that's why Apple created bitcode to try to help with that.

Oh, have you ever worked at a traditional tech company? Bureaucracy, contracts, silly guidelines often lead to very stifled vendor lock in. Even if the whole tech department says. "hey we should go AMD" giving every manager involved enough time to make themselves important over the issue can cost you half a decade.

At a pure tech company, moves across architectures and platforms is an absolute non-story unless the company is broken: Choose and implement hardware to achieve desired goals at the right price points. Reassess as the industry changes.

That service you are talking to might be running on a Xeon Windows Server instance on vmware, or a Neoverse N1 Linux instance over KVM. Whatever was right. Cloudflare specifically say that they try to operate this way, and probably have since day, as does any competent tech company.

It's more the tech-as-a-cost-center company that has such restrictions. The accounting shop or traditional bank where "We use X, Y and Z! We're an Intel and Dell company!", where the desktops are all running Windows 7.

We should maybe define pure and traditional better, but yes, I was mostly thinking companies which (heavily) rely on tech, but more as a commodity driving their actual business, and never tried to push the tech to its limits.

Incapsula?

Title is "Technical Details of Why Cloudflare Chose AMD EPYC for Gen X Servers"

The intriguing thing is that the guy who submitted the story is the Cloudflare CTO...

Blog post is by Nitin Rao, Head of Global Infrastructure at Cloudflare.

Submission and editorialized title change is by John Graham-Cumming, CTO at Cloudflare.

Not sure what to make of that.

jgrahamc is more active on HN, and he seems to keep tabs on all Cloudflare discussions and answers questions when asked. Don't see what's odd about it. Haven't noticed Nitin here.

It's not odd that he submitted it. It's odd that he changed the title, given the HN guidelines against changing titles.

https://news.ycombinator.com/newsguidelines.html

We've reverted the title to that, in accordance with the site guidelines.

Submitted title was "Why Cloudflare abandoned Intel for AMD for its latest servers".

Let's put it this way.

I'd rather program single core applications on my i5-4690k than any other ryzen series chips including threadrippers with 12+ cores.

I have used two threadrippers for more than 6 months.

Linux still runs faster on i5-4690k which is 6 years old

Odd, I switched from that processor to the AMD 2700 last month and I'm enjoying the legroom.

Reply on news.ycombinator.com