Hacker News Reader

Comment by saas_co_de | Hacker News Reader

> Their failed 10nm shrink has forced product delays

This is the most important thing. AMD has already made the switch to Multi Chip Modules which makes it much much easier to produce chips for 10nm (what TSMC/GF/Samsung call 7nm).

Right now Intel cant even make a dual core low speed mobile chip on 10nm. How are they going to make a giant 30+ core server processor? This is extremely bad news for them that has not been fully realized by the markets because they have faith that Intel will figure it out, but they may not.

AMD may ship 7/10nm server chips before Intel and Intel may never ship them before switching to MCM themselves.

MCM is as revolutionary as AMD64 was but most people dont realize yet how important it is and how much of an advantage AMD has because of leading with it.

View on HN

AMD may be shipping server CPUs as MCMs while Intel isn't, but that doesn't mean AMD has a meaningful lead. Intel's Embedded Multi-Die Interconnect Bridge (EMIB) [0] technology is far more advanced than any MCM tech AMD has available, to the point that AMD is using Intel's EMIB to connect an AMD GPU to its DRAM in an upcoming Intel product [1].

When Intel gets around to making a multi-die CPU using EMIB, it'll blow away AMD's EPYC in terms of bandwidth between dies.

[0] https://www.intel.com/content/www/us/en/foundry/emib.html

[1] https://www.anandtech.com/show/12003/intel-to-create-new-8th...

> AMD is using Intel's EMIB to connect an AMD GPU to its DRAM in an upcoming Intel product

It is Intel's CPU using AMD's GPU. I am not sure how this is evidence that EMIB is superior tech. Of course Intel is going to use their own tech wherever possible.

> When Intel gets around to making a multi-die CPU using EMIB, it'll blow away AMD's EPYC in terms of bandwidth between dies.

Do you have any bandwidth numbers on EMIB to support this? There are other sources saying that EMIB is crap and doesn't interface with HBM without using a bridge provided by TSMC. [1]

[1] https://semiaccurate.com/2017/12/19/intels-claims-fpgas-hbm-...

Make sure to take SemiAccurate's name seriously.

That article doesn't say EMIB is broken, it just says that the FPGA product Intel/Altera announced wasn't actually natively designed to use EMIB to interface with HBM. This is not relevant to a discussion of whether two Intel CPU dies could be connected over EMIB.

We do know that the Intel CPU with an AMD GPU on-package will have a much faster interconnect between the dGPU and its HBM2 over EMIB than AMD's EPYC and Threadrippper manage between CPU dies with a conventional MCM. It's also faster than Intel's old Crystalwell parts that used a conventional MCM to add an eDRAM L4 cache to a consumer CPU.

The limits of conventional MCM packaging are well-known across the industry and are a problem for everyone who's trying to use HBM-style DRAM or otherwise make high-speed inter-die links. Full-scale silicon interposers with TSVs are really expensive and would not have been adopted in high-end GPUs and FPGAs over the past several years if there was an alternative. Likewise, AMD wouldn't have adopted EMIB for their project with Intel if conventional MCM packaging were sufficient, because EMIB is still more expensive than not using two kinds of interconnects within the package.

> Likewise, AMD wouldn't have adopted EMIB for their project with Intel if conventional MCM packaging were sufficient

Once again, you are talking about an Intel CPU, so what you are saying makes no sense. It is not AMD's project. It is Intel's CPU that happens to use an AMD component.

> Once again, you are talking about an Intel CPU, so what you are saying makes no sense. It is not AMD's project. It is Intel's CPU that happens to use an AMD component.

The only EMIB in that product is between AMD's GPU and that GPU's DRAM; the Intel CPU die in that product is not using EMIB. The fact that the AMD GPU is using EMIB is significant—it's a somewhat external validation that EMIB has value as an alternative to conventional MCM packaging or large scale silicon interposers, both of which AMD uses for products of their own.

If EMIB wasn't saving money while offering sufficient performance, then Intel would have integrated the AMD GPU into this product using the off-the-shelf silicon interposers that AMD's own products use to connect the GPU to its HBM DRAM.

I don't understand. How does the CPU talk to the GPU on that product? What bridge the GPU die to the CPU die? Sounds like you are saying the CPU doesn't talk to the GPU over EMIB.

> Sounds like you are saying the CPU doesn't talk to the GPU over EMIB.

Correct. There's a conventional link through the package substrate carrying the PCIe signals from the CPU to the GPU. The only EMIB on that product is between the GPU and its DRAM. That's why the GPU and its DRAM are adjacent, while the CPU is about 1cm away from them: https://images.anandtech.com/doci/12003/intel-8th-gen-cpu-di...

Intel would need to redesign the CPU die to accommodate an EMIB connection to the CPU. As it stands, an EMIB link would probably get in the way of lots of contacts that need to go to something other than the GPU, and EMIB wouldn't be worth the trouble for a relatively narrow and slow link like PCIe. The point of EMIB is to enable very wide and fast links between dies, so that inter-die communication is almost as fast as communication across a single die.

> If EMIB wasn't saving money while offering sufficient performance, then Intel would have integrated the AMD GPU into this product using the off-the-shelf silicon interposers that AMD's own products use to connect the GPU to its HBM DRAM.

That is a nice claim, based on no evidence.

Intel spends billions subsidizing non-competitive products in order to try to build market share all the time. (mobile, ultrabook, etc, etc)

Unless you can present any evidence to the contrary (still no bandwidth numbers) it is just as likely that this product has EMIB because they couldn't get anyone else to use it and they need some shipping products with the tech to try and sell it.

> Intel spends billions subsidizing non-competitive products in order to try to build market share all the time.

The concept of market share does not apply to EMIB. It's an internal implementation detail, and Intel gains nothing from using EMIB in one of their own products over a cheaper solution that would still result in the end product being an Intel-branded part.

The only motivation that Intel could possibly have to use EMIB as any kind of loss-leader would be to entice third-party foundry customers to use Intel for the sake of EMIB. But attracting foundry customers is clearly not a priority for Intel, and Intel's primary goal with EMIB is to produce Intel products that may incorporate some third-party silicon, not to produce silicon to be incorporated into third-party products.

> Intel may never ship them before switching to MCM

Fantastic claims require fantastic evidence.. history shows that there are many ways of fixing or designing-around manufacturing yield problems. Do you have any actual evidence that their 10nm fabs will never yield a competitive monolithic server part in a financially-viable way?

> MCM is as revolutionary as AMD64

I would disagree somewhat, as AMD64 clearly benefits every customer with very little downside, while it is fair to say that MCM CPUs can create or exacerbate irritating performance problems for the customer by being "much more NUMAed". [All else equal, most customers would prefer a monolithic part, or 2 NUMA domains instead of 8-16].

Not quite. 64 bit ran behind 32 bit for a long time after introduction. The rule of thumb was to ask whether more ram or more speed was important. 64bit felt slower to people for a long time. (I swear I can still see the difference 10 years later.) 64 bit didn’t really pull away until 8 gig became common. Something similar may hold for MCM, I don’t know about mcm to speak to that.

To clarify my point: an AMD64 CPU could still run in 32-bit mode, if the customer preferred. Much later, the x86_32 ABI work also retained small pointer sizes while providing additional benefits. Whereas there is no way for a customer to fully "unNUMA" an MCM CPU.

We saw substantial performance improvements from the increased register count of AMD64 when I lead the porting effort of a video editing/compositing suite back in the early days.

Because our engine took advantage of memory-mapped unnamed file handles to cache frames, we didn't really need the extra address space of larger pointers. I was able to exhaust system memory by manually managing whether or not buffers were mapped into our address space.

Though it had been made for something else, I lucked into being able to use that same engine for interprocess legacy plug-in support. AMD64 support also wasn't necessary to get more physical memory support out of the chips of the era. Operating systems supported 36 and 40 bit physical address spaces, and applications could use higher (would-be negative) addresses by indicating support for it. Those wanting to really cut loose could turn to manual mapping. (See: PAE, LAA)

In the end, we shipped AMD64 support because there were substantial performance wins. And, yes, I profiled every possible software/hardware combination to determine where gains and losses were coming from.

I'm not saying that things were universally faster/better. RIP-relative addressing, though generally quicker than absolute 64-bit, is a headache, and the loss of 80-bit floating point intermediates broke some rare code (which would have broken in strict mode, anyway). I'd love to see some evidence of slow-down now, but, until then, I'm unconvinced. I had a dev bring me "evidence" of huge performance differences between ARMv7 and ARMv8 a couple of years ago (and there are some if you dig). I asked him if he had compiled with or without thumb in the linked modules.

He let me know of his new results a few days later, a little red-faced.

I'm not saying that your detection of slowness is the same thing, but many aspects of performance are measurable (and measured). If it's slower, we can probably measure just how much.

Yes, if someone said anything about video editing I would suggest a 64 bit system.

When you say "MCM is as revolutionary as AMD64", does that mean Debian and RedHat and Windows and macOS, and all 3rd party applications, have to be recompiled for a new architecture? Or is it simply an incremental speed-up of existing systems?

I'm led to believe that if we're going to recompile The World as well as build new JIT javascript++ compilers, it will be for ARM64?

> does that mean Debian and RedHat and Windows and macOS, and all 3rd party applications, have to be recompiled for a new architecture? Or is it simply an incremental speed-up of existing systems?

Neither. Multi-chip modules simply make it more economical to manufacture large processors. When you can make a 32-core CPU by connecting two 16-core silicon dies together, you'll have higher yields than if you try to make a monolithic 32-core die. There isn't necessarily any consequence visible to software, with the possible exception of having multiple NUMA nodes per physical socket.

So not an architectural paradigm shift, then - a mostly "invisible" incremental CPU improvement instead?

It's a pretty big increment that Intel has been unable to or unwilling to advance to. The message here is that if Intel drags their feet long enough then it'll take that much longer to catch up -- or, more to the point, they'll be forced to take more drastic (read: expensive) measures to catch up.

Intel has produced multi-die CPUs in the past. They aren't doing it now because their fabrication advantage allows them to make bigger monolithic CPU dies than AMD can afford, so Intel doesn't have to incur the performance penalties that AMD's multi-die CPUs suffer from. At the same time, Intel has been very publicly preparing for the day that it makes sense for them to once again make multi-die CPUs, by making sure they have the technology to do it without the performance hit that is currently necessary. There's no heel-dragging from Intel on this issue.

Once Intel's fab advantage is actually eliminated, they'll be forced to make multi-die CPUs to compete against AMD's multi-die CPUs. But every indication is that Intel will be ready to do multi-die better than AMD is doing multi-die. How much better is up for some debate, but it does look like Intel will have a leg up on EPYC's biggest weakness.

Multi-Chip-Module (Epyc) is just 4 separate dies wired together. (As Intel would derisively point out if is basically 4 Ryzen desktop chips glued together).

The advantage of this architecture is that smaller chips are easier to manufacture because the probability of a random defect rendering the chip useless is lower and when a chip is bad it is not as great of a loss.

Epyc chips run all existing software. There is extra work involved with Non-Uniform Memory Access (NUMA) where if you want software to actually use 32 or 64 cores simultaneously it has to be purposefully designed to take into account the architecture but that is not really any different than developing for multi-processor systems in the past.

The point is that we have hit the ceiling on frequency scaling, which means more cores, and now we have hit the ceiling on core count scaling, so now it is going to be more CPUs, and the best way to do more CPUs is with MCMs.

MCM in itself is old: Xenos in XBox 360 was MCM, IBM's POWER chips have been MCM's in many generations, Intel's Pentium Pro was MCM 1995, etc. Going back to the 80's/70's.

MCM has been relatively unpopular because of cost and that's why most of its applications could justify it only to put chips from different silicon proceseses in same package (cache or EDRAM + CPU usually). Except for some high end chips like the POWER stuff.

The interesting bit is how and why AMD put logic chips in low cost MCMs. Probably involves the time or $$ budget to make and validate additional high end variations of the silicon.

> 10nm (what TSMC/GF/Samsung call 7nm)

What’s the reason for this? From my understanding of numbers, 10 is not 7

They are measuring different things and use different transistor designs.

well yeah not really. it's a bit of a long read but it comes down to whom you ask. https://www.extremetech.com/computing/246902-intel-claims-th...

"10nm" and "7nm" are marketing terms that are not directly comparable across foundries.

How? I thought the number was the size of the smallest transistor?

The numbers are almost pure marketing, these days.

From https://en.wikipedia.org/wiki/7_nanometer#7_nm_process_nodes

> The naming of process nodes by different major manufacturers (TSMC, Intel, Samsung, GlobalFoundries) is partially marketing driven and not directly related to any measurable distance on a chip – for example TSMC's 7 nm node is similar in some key dimensions to Intel's 10 nm node.

Reply on news.ycombinator.com