AMD's Strix Halo under the hood
chipsandcheese.com
chipsandcheese.com
Aren't there enough syllables out there to invent a combination which doesn't collide with your own board partners?
AMD Ryzen AI MAX 300 is the product name. This continuing to use the code name.
Did AMD not know?
Or did AMD know and not care?
AMD is captured.
AMD's marketing names for especially their mobile chips are just so deliberately confusing that it makes way more sense for press and enthusiasts to keep referring to it by its internal code name than whatever letter/number/AI nonsense AMD's marketing department comes up with.
In addition there's a interest in having a lot of memory for LLM acceleration. I expect both CPUs to get more LLM acceleration capabilities and desktop pc memory bandwidth to increase from its current rather slow dual channel 64bit DDR5-6000 status quo.
We're already hearing the first rumors for Medusa Halo coming in 2026 with 50% more bandwidth than Strix Halo.
GPU:s have existed about 30 years. Embedded ones for 20 years or so? Why are the embedded GPU:s always so stunted?
Why are the embedded GPU:s always so stunted?
Before someone tells me consoles using iGPUs, keep in mind that consoles use GDDR as its main system memory which has slow access times for the CPU. In a non-console, CPU performance is important. GDDR is also power hungry so they can't be used as the main system RAM in a laptop form.
It is the thermal envelope that defines pretty much everything nowadays. Without active management of it chips would die a heat death very fast. Which also means chips are designed with a certain chip external heat management in mind. The more heat you can get out of a system and away from a chip, the more powerful you can design these things. And game consoles do have active cooling, i.e. they sit between desktop PCs and thin laptops, probably sharing the thermal handling capacity with larger gaming laptops, if anything.
On Desktop, upgradability is very popular and obviously the returns from the cooling on discrete GPUs are immense. With GPU dies costing so much, due to their size and dependency on TSMC, pushing the faster but hotter is probably a cost effecient compromise.
On Laptops with APUs, you currently ususally give up upgradeable memory - the fastest LPDDR is only soldered on (today), and the fastest solution would be on-die memory for bandwith gains that only really Apple is doing.
Marketing wise, low core count Laptops appear to be hard to sell. Gaming laptops seem to ship with more cores than the desktop you would build - the CPU appears out-specced. I think this is because CPUs are cheaper, but that means a high-end APU would also need large CPU to compete. Now you've got a relatively unbalanced APU, with expensive hot CPU and relatively hot iGPU crammed in a small space - cooling is now tricky.
This is going to be compared with cheap RTX 4060 laptops - and generally look bad by comparison. I think what's changing now to narrow the gap is Handhelds, and questionable practices from Nvidia.
The Steam Deck kicked big OEMs into requesting AMD for large APUs.
Nvidia seems to have influence on OEM AMD Laptops - Intel CPU and Nvidia GPU for years now seem to ship first, in larger quantities, and get marketing push despite CPU arguably being worse.
Intel despite their issues seem to raising the iGPU bar too - their Desktop GPU investment seems to be paying off, and might be pressuring AMD to react.
If you want a good value for your money you need modularity and competition for the modules. If it's a one package deal the companies will charge so much that it curbs secondary markets that could be created, which could add value to the product.
Because gpu want a lot of silicon. 5080 is 300 mm^2. Meanwhile ryzen 9xxx is 50 mm^2.
Meanwhile CPU wants to use that wafer space for themselves. And even if you use 100% of the wafer space for GPU you will have a small gpu and no cpu
Not really closer. igpus got good enough to kill the low the discreet market basically entirely, but they haven't been "getting closer" to discreet cards. Both CPU SoCs and discreet GPUs have access to the same manufacturing nodes and memory technologies, and the simple physical reality is that they can just be bigger and use more power as a separate physical entity, along with memory better optimized for its workloads.
Historically those cards had narrow memory bus and about a quarter or less video memory of high end (not even halo) cards from the same generation.
That narrow memory bus puts their max memory bandwidth at a comparable level to desktop DDR5 with 2 DIMMs. At the same time quarter of high end is just 4GB VRAM which is not enough for low details for many games and prevents upscaling/frame gen from working.
From manufacturing standpoint low end GPUs aren't great either - memory controllers, video output and a bunch of other non-compute components don't scale with process node.
At the same time unified memory and bypassing PCIE benefits igpus greatly. You don't have to build an entire card, power delivery, cooler - you just slightly beef up existing ones.
tl;dr; sub-200 dollas GPUs are dead and will be replaced by APUs. I won't be surprised if they will start nibbling at lower mid-range market too in the near future.
that already happened like 5+ years ago. The GT 1030 never got an update, so Nvidia hasn't made an entry level GPU since. Intel kinda did with ARC, but that was almost more a dev board
Probably, not. Because it need dedicate channel on hardware level.
- GPU are mostly for streaming applications with large data blocks, so usually, CPU cache architecture is too different from GPU to simply copy (move) data, plus, they are on different chiplets, and dedicate channel means additional interface pins on imposer which are definitely very expensive.
So, when it is possible to make SoC with dedicated channel CPU<->GPU (or between chiplets), but usually it used only on very expensive architectures, like Xeon, or IBM Power, and not used on consumer products.
For example on older AMD products with APU, usually, graphics core have priority over the CPU to access unified RAM, but CPU cache don't have any additions to handle shared with GPU memory.
On latest IBM Power and similarly on Xeon, invented shared L4 cache architecture, where blocks of extremely huge L4 (near to 1Gb per socket on Power, as I remember somewhere about 128Gb on Xeon), could be assigned programmatically to exact core(s) and could give extremely high performance gain for applications running on these cores (usually these things very beneficial for DB or something like zip compressing).
Added: example difference CPU cache to GPU, for CPU usual size of transaction is less than 64bits, may be current 128..256bits but this is not common on consumer hardware (could be on server SoC), just because many consumer applications are not optimal to use large blocks, but for GPU normal to use 256..1024bits bus, so their cache definitely also have 256bits and larger blocks.
Mean, in CPU could cut any core and it will work completely separated without other cores.
In GPU, typical, have blocks for example 6x CUs, which have one pipeline for all, and this is how they achieve thousand CUs or more. So, all CUs basically run same program, in some architectures could make limited independent branching with huge penalties on speed, but mostly just one execution path for all CUs.
Very similar to SIMD CPU, even some GPUs was basically SIMD CPUs with extremely wide data bus (or even just VLIW). So, GPU cache sure optimized for such usage, it provide buffer wide enough for all CUs on same time.
So, when CPU access GPU memory, CPU just directly access RAM via system bus, but not trying to check GPU cache. And yes, this mean, could be large delay between GPU write cache and data actually delivered to RAM and seen by CPU, but probably, smaller than on discrete GPU on PCIe.
I sure hope so. We could use a new board form factor that omits the GPU slot. Although my case puts the power connector and button over that slot on the back so it's not completely wasted, but the board area is. This has seemed like a good idea for a long time to me.
This can also be a play against nVidia. When mainstream systems use "good enough" integrated GPUs and get rid of that slot, there is no place for nVidia except in high-end systems.
Below the mini-ITX format with a GPU slot, there are 3 standard form factors that are big enough for a full-featured personal computer that is more powerful than most laptops: nano-ITX (120 mm x 120 mm, for 5" by 5" cases; half the area of mini-ITX), 3.5" (from the size of the 3.5 inch HDDs, approximately the same area with nano-ITX, but rectangular instead of square) and the 4" x 4" NUC format introduced by Intel.
With a nano-ITX or 3.5" board you can make a computer not bigger than 1 liter that can ensure a low noise even for a 65 W power dissipation for the CPU+iGPU and that can have a generous amount of peripheral ports, to cover all needs.
Keeping the low noise condition, one could increase the maximum power-dissipation to 150 W for the CPU+iGPU in a somewhat bigger case, but certainly still smaller than 2.5 liter.
I expect that we will see such mini-PCs with Strix Halo, the only question is whether their price would be low enough to make them worthwhile.
The fabrication cost for Strix Halo must be lower than for a combo of CPU with discrete GPU, but the initial offerings with it attempt to make the customer pay more for the benefit of having a more compact system, which for many people will not be enough motivation to accept a higher price.
One of (many) factors that were holding back this form factor was the gap in iGPU/GPU performance. However with the frankly total collapse of the low end GPU market in the last 3-4 years, there's a much larger opening for iGPUs.
I also think that within the gaming space specifically, a lot of the chatter around the Steam Deck helped reset expectations. Like if everyone else is having fun playing games at 800p low/medium, then you suddenly don't feel so bad playing at maybe 1080p medium on your desktop.
Pay for memory once, and avoid all the copying around between CPU/GPU/NPU for mixed algorithms, and have the workload define the memory distribution.
In a next iteration AMD could look into doubling or quadrupling the memory channels and GPU die area like as Apple has done. AMD is already a pioneer in the chiplet technology Apple is also using to scale up. So there's lots of room to grow for even higher costs.
I wish Chips and Cheese would clean up transcripts instead of publishing verbatim. Maybe I'll use the GPU on my Strix Halo to generate readable transcripts of Chips and Cheese interviews.
Although I appreciate the drive for small profile I wonder where the limits are if you put a big tower cooler onto it, seeing as the broad design direction is for laptops or consoles I doubt there's too much left on the table. I think that highlights a big challenge - is there a sizeable enough market for it, or can you pull in customers from other segments to buy a NUC instead. You'd need a certain amount of mass manufacturing with a highly integrated design to make it worthwhile.
I've never understood the hype for NUCs for non-office settings. You can make SFF builds that are tiny and still fit giant GPUs like the RTX 3090 /4090, say less for something like a 4080 Super. And then you can upgrade the GPU and (woe is you) CPU later on. Although a high-end X3D will easily last you 2-3 GPU generations.
https://github.com/phkahler/mellori_ITX/blob/master/images/m...
It's currently got a 5700G - Zen 3 in it and 64GB RAM. I'd like the next one to hang on the back of a monitor or TV via the standard mounting holes.
They're great shop computers, for example. It fits anywhere, it doesn't wreck your energy bill if you leave it on all the time, and the nicer ones have juuust enough oomph to load up 3D CAD files or whatever else you might need.
I also really want an update to typical form factors and interconnects of desktop computers. They've been roughly frozen for decades. Off the top of my head:
- Move to single-voltage power supplies at 36-57 volts.
- Move to bladed power connectors with fewer pins.
- Get rid of the "expansion card" and switch to twinax ribbon interconnects.
- Standardize on a couple sizes of "expansion socket" instead, putting the high heat-flux components on the bottom side of the board.
- Redesign cases to be effectively a single ginormous heatsink with mounting sockets to accept things which produce heat.
- Kill SATA. It's over.
- Use USB-C connectors for both power and data for internal peripherals like disks. Now there's no difference between internal and external peripherals.
Framework asked AMD if they could use CAMM for their new Framework Desktop.
AMD actually humored the request and did some engineering, with simulations. According to Framework, the memory bandwidth on the simulations was less than half of the soldered version.
This completely defied the entire point of the chip - the massive 256 bit bus ideal for AI or other GPU-heavy tasks, which allows this chip to offer the features it does.
This is also why Framework has apologized for non upgradability, but said it can’t be helped, so enjoy fair and reasonable RAM prices. Previously, it had been speculated that CAMM had a performance penalty, but Framework’s engineer on video saying it was that bad was fairly shocking.
CAMM is touted as being better than DIMMs when it comes to signal integrity and possible speed. Soldered of course beat any socket, in-package beats any soldered RAM, and on-die beat any external component.
That AMD Strix Halo is unable to maintain signal integrity for any socketed RAM is a Strix Halo problem, not a socket problem. They probably backed themselves a bit into a corner with other parts of the design sacrifying tolerances on the memory side, and it's a lot easier to push motherboard design requirements than redoing a chip.
If this wasn't a Strix Halo issue, then they would have been able to run with socketed memory with a lower memory clock. All CPUs, this one included, has variable memory clocks that could be utilized and perform memory training as even the PCB traces to the chip cause significant signal degradation.
I suspect the earlier comment about "Half the performance with CAMM" is likely at iso-power, but that might still be a pretty big dealbreaker.
CPU's and GPU's reclock extremely fast to my knowledge, but what we're talking about isn't dynamic reclocking, just limiting the max clock as suitable to the system design.
We already see this when we have laptop silicon that run faster clocks when using soldered RAM compared to when the same silicon is using socketed counterparts.
That this wasn't an option probably mean that they're either far too close to the limit, or unwilling to allow a design that runs below max speed.
LPDDR6X is designed for it, and an use CAMM2.
Why? And why not 12V? Please be specific in your answers.
> - Get rid of the "expansion card" and switch to twinax ribbon interconnects.
If you want that, it's available right now. Look for a product known as "PCI Express Riser Cable". Given that the "row of slots to slot in stiff cards" makes for nicely-standardized cases and card installation procedures that are fairly easy to understand, I'm sceptical that ditching slots and moving to riser cables for everything would be a benefit.
> - Kill SATA. It's over.
I disagree, but whatever. If you just want to reduce the number of ports on the board, mandate Mini SAS HD ports that are wired into a U.2 controller that can break each port out into four (or more) SATA connectors. This will give folks who want it very fast storage, but also allow the option to attach SATA storage.
> - Use USB-C connectors for both power and data for internal peripherals like disks.
God no. USB-C connectors are fragile as all hell and easy to mishandle. I hate those stupid little almost-a-wafer blades.
> - Standardize on a couple sizes of "expansion socket" instead...
What do you mean? I'm having trouble envisioning how any "expansion socket" would work well with today's highly-variably-sized expansion cards. (I'm thinking especially of graphics accelerator cards of today and the recent past, which come in a very large array of sizes.)
> - Redesign cases to be effectively a single ginormous heatsink with mounting sockets...
See my questions to the previous quote above. I currently don't see how this would work.
Higher voltages improve transmission efficiency, in particularly for connectors, as long as sufficient insulation is easy to maintain. Datacenters are looking at 48V for a reason.
Nothing comes for free though, and it makes for slightly more work for the various buck converters.
> God no. USB-C connectors are fragile as all hell and easy to mishandle. I hate those stupid little almost-a-wafer blades.
They are numerous orders of magnitude more rugged than any internal connector you've used - most of them are only designed to handle insertion a handful of times (sometime connectors even only work once!), vs. ten thousand times for the USB-C connector. In that sense, a locking USB-C connector would be quite superior.
... on that single metric. It would be ridiculously overcomplicated, driving up part costs when a trivial and stupidly cheap connector can do the job sufficiently. Having to run off 48V to push 240W and have no further power budget at all also increase complexity, cost and add limitations.
USB-C is meant for end-user things where everything has to be crammed into the same, tiny connector, where it does great.
The idea is that USB-C would be the "small widget" connector for things like large form factor disks (not m.2), optical drives (as much as those still exist), etc. Heavy iron things would use the bladed connector for power and the twinax ribbon for data. PCIe is great.
It would also not sadden me tremendously to have CPU+RAM be a single device, be it on the CPU substrate or as a system-on-module. That flavor of upgradeability isn't worth losing a ton of performance. I'm sure tons of folks would hate it.
Having a desktop CPU with integrated 64-128GB of RAM would probably justify the lack of "upgradability" with sheer performance and power improvements. There was never much upgradability there anyway - you could start by underspec'ing and then later fill it up to the (surprisingly not that high) limit, but what you saved by not buying all the RAM up front is probably less than the extra cost for supporting it at all. Paying more than the cost of RAM to have the flexibility of not buying RAM.
(Phones have 24GB of RAM nowadays, a PC should not have any less than 64GB of RAM as base tier.)
Have you run the numbers on cost for USB-C plugs, ports, and cabling vs. what you're wanting to replace? My strong expectation is that you're proposing something that's substantially more expensive than what it's replacing.
It's also a plan that places one or more cables with electrically-conductive ends into a chassis with components that react... poorly to unplanned contact with electrically-conductive materials. Do remember that PC internals have been designed to work with cables that have been carefully designed to have electrically non-conductive ends when not plugged in to their intended socket.
I asked you a bunch of other questions that I'd legit like the answers to. I wouldn't have bothered asking the questions if I wasn't interested in the answers.
What datacenters are doing has little bearing on what's worth the replacement and reengineering cost in a SOHO environment. Datacenters have multiple orders of magnitude more power they need to deal with than a SOHO user.
Also, given the long-standing existence of the 80 Plus program, I'm also pretty suspicious of claims that switching to HVDC inside the computer chassis will have an effect on a end-user's power bill that's not washed out by the increased expense of the replacement components. Will it? Do you have the numbers?
> (sometime connectors even only work once!)
I've been building PCs for decades. While I totally believe that there are such connectors, I've never been so unlucky as to encounter one.
> They are numerous orders of magnitude more rugged than any internal connector you've used.
Some? Sure. Any? Hell no. For example: You cannot honestly claim that a bigass ATX power connector is less resistant to shear force than the tiny wafer that is the USB-C connector. I've had many USB-C cables (and some ports!) fail at the connector end due to stresses that ATX cables handle just fine.
But yeah, I totally agree that -say- SATA connectors, or "freestanding" front-panel pin connectors are far more vulnerable to shear and torque than USB-C connectors.
I guess you're not specifying tolerances here, so it'll be difficult to have a real conversation... but I've been watching a bunch of teardowns of video cards that have been released in the past few years, and I'm seeing substantial difference in PCB sizes and component placement location.
Even putting that aside, OP hasn't provided any details on what exactly they were proposing... which is a damn shame, as I was interested in hearing them.
Does anyone know?
Too bad there isn't a full PCIe (might not be enough bandwidth left) :(.
$1,299 (64GB) and $1,599 USD (128GB) for the motherboards. Yikes, but I get why.
I wouldn't be surprised if Minisforum also offers a motherboard.
You shouldn't need another GPU to do prompt processing for Strix Halo since the biggest model it can realistically run is a 70B model. Prompt processing isn't going to help much because it has a good enough GPU but its memory bandwidth is only 256GB/s (~210 GB/s effective).
What a... strange statement. How did you get to that conclusion?
I'm wondering how do you get to such a conclusion?
but is a competitor to the $4000 Mac Studio with 96 GB. The slow memory bandwidth is a bummer, though.
$2000 M4 Pro Mini is more of a direct comparison. The Mini only has 64GB max ram but realistically, a 32B model is the biggest model you want to run with less than 300 GB/s bandwidth.
Let's be realistic here.
The compute, bandwidth, capacity (if 128GB) are completely imbalanced for Strix Halo. M4 Pro with 64GB is much more balanced.
2 to 3 tokens per second was actually probably fine for most things last year.
Now, with reasoning and deep searching, research models, you’re gonna generate 1000 or more tokens just as it’s talking to itself to figure out what to do for you.
So everyone’s focused on how big a model you can fit inside your ram, the inference speed is now more important than it was.
The thinking models really hurt. I was happy with anything that ran at least as fast as I could read, then "thinking" became a thing and now I need it to run ten times faster.
I guess code is tough too. If I'm talking to a model I'll read everything it says, so 10-20 tok/s is well and good, but that's molasses slow if it's outputting code and I'm scanning it to see if it looks right.
I still agree, though.
Feedback: That 4x slot looks like it's closed on the end. Can we get an open-ended slot there instead so we can choose to install cards with longer interfaces? That's often a useful fallback.
Gen 4 x4 or gen 5 x4? I saw that gen 5 x4 results in maybe a 3% decrease in 5090 performance compared to gen 5 x16.
I realize pairing this with a 5090 is a bit ridiculous, but... is it?
https://www.techpowerup.com/review/nvidia-geforce-rtx-5090-p...
This solves those problems but apparently uncovers a new one.
Surely the vast majority of laptops sold in the last five years don't have unified memory yet.
- The CPU/GPU memory are shared (does not have to be dedicated to be used by either).
- You don't need to copy data in memory to move it between the CPU/GPU.
- It still uses separate caches for the CPU & GPU but the two are able to talk to each other directly on the same die instead of an external bus.
But these have long been true of traditional APUs/iGPUs, not new changes. I did even see some claims Apple put the memory on die too and that's what makes it unified but checking that it seems to still actually be "on package", which isn't unique either, and it wouldn't explain any differences in access patterns anyways. I've been particularly confused as to why Strix Halo would now qualify as having Unified Memory when it doesn't seem anything is different than before, save the performance.
If anyone has a deeper understanding of what's new in the Unified Memory approach it'd be appreciated!
Reading about HSA again (it's been many years) it seems like ARM was one of the ones originally part of defining it back then and I wonder if Apple actually did anything different on top this at all or just branded/marketed the use of what was already there.
It seems there are 3 key things which distinguish the modern "unified memory architecture" from its predecessors:
1. Pointer passing instead of buffer copying
2. Separate bus for memory access alongside the PCIe bus
3. No partitioning of RAM into exclusive CPU vs. GPU areas
AMD seems to have delivered the first hardware meeting these criteria about 8-9 years ago, beating Apple to the punch by a couple years. However, AMD's memory bandwidth can be quite a bit behind Apple's. The M1 Pro handily beats anything AMD had out at the time (200 GB/s vs. 120 GB/s), and the M1 Max has double that bandwidth.
Framework (I believe) made one of these into a purchasable desktop.
They tend to come out much slower than the laptop chips, or the "CPU-only" desktop chips.
This is one of the more recent examples: https://www.amd.com/en/products/processors/desktops/ryzen/80...
So why build powerful laptops? Simple: people want powerful laptops. Remoting to a desktop isn't really a slam dunk experience, so having sufficient local firepower to do real work is a selling point. I do work on both a desktop and a laptop and it's nice being able to bring a portable workstation wherever I might need it, or just around the house.
I'd work from my workstation at all times if I could. Tramp is alright, but not too fast and fundamentally can't make things transparent.
The mini-PC market (which basically all uses laptop chips) seems pretty robust (especially in Asia/China). They've basically torn out the bottom of the traditional small form factor market.
It won over OpenCL, because it is a polyglot ecosystem, with first tier support for C, C++, Fortran, and Python (JIT DSL), plus several languages that have toolchains targeting PTX, the IDE integration, graphical debugger, compute and graphical rendering libraries.
All of the above AMD and Intel could have provided for OpenCL, but never did when it mattered, not even after SPIR was introduced.
Now they finally have GPGPU support for Fortran, C++, Python JIT DSLs, but a bit too late to the party, because contrary to NVidia it isn't like those tools are available regardless of the card.
You don't need all the fancy stuff, but OpenCL (and even more so Vulkan) are too complicated when all you want to do is some gpu number crunching.
Being able to write a kernel with something that looks like C. Having pointers on gpu and cpu and being able to call these kernels somewhat conveniently (like CUDA C) would be a great starting point.
And the M2 Max has a memory bandwidth of 400GB/s.