Intel Gaudi 3 the New 128GB HBM2e AI Chip in the Wild

Ironically enough, since NVIDIA bought Mellanox, it's likely that the best documented route to get ROCE v2 going is with switches purchased from NVIDIA...

Edit: and yes, it's RDMA over ethernet https://docs.habana.ai/en/latest/Gaudi_Overview/Gaudi_Archit...

What does?

They are 24x200G, which is already outdated. Everything we are doing is currently 400G (via 8xCX7 cards running in ethernet mode) and 800G at the spine. 800G NICs, which will come with PCIe6 next year and cuts the number of connections down.

What I'd prefer is the connection is through the UBB/OAM baseboard, such that you have PCIe connections. Look into what GigaIO and Liqid are doing. There is a 3rd option that is even cooler than those two, but I don't want to mention it here. ;-)

This is a big part of why they are generally available. Building on top of AMD gear is currently more difficult than Nvidia. It's unfortunate, but the evolution of/from Cuda when AMD had little in this space gave intel a massive jump-start. As a dev it's way easier to pick up various stacks that work with nvidia gear and not have to worry anywhere near as much about what's underneath.

I’m aware.

In the keynote Nvidia is quoting FP4 performance for Blackwell specifically. Frontier and every computer that has ever hit the supercomputer top 500 is measuring fp64. This is why I qualified my statement above by saying if you’re willing to compare apples and oranges.

But I think this is where things open up to debate (and/or personal interpretation). My view is, if all you care about is AI workload (Specifically LLMs), then you really are seeing a full two orders of magnitude improvement. And if we are trying to get a feel for the space, and what that even means, then there really is nothing else to compare Blackwell to other than frontier, in terms of scale alone.

Above I say “one or two orders of magnitude”, the “one order of magnitude” is a value I got by taking Blackwell’s FP4 values and dividing by 16, to try to conceptually convert back to a value that can be compared to fp64. — I’m perfectly happy to admit that all I’m really capable of doing here is using estimates to compare a completely new thing (the new compute architecture of Blackwell) to what came before it (the majority of the history of the supercomputing 500). But that’s okay, because that’s my only goal.

—-

Believe me, I get it. Blackwell will never post a linpack score. Blackwell will never run big scientific computing jobs like… global weather simulations.

But as computer science nerds, how could we not be excited to see such a big move: changing the computer architecture to better suit a specific compute workload (LLMs)?

I know Bitcoin has its asic processors, but to me Blackwell’s mission statement of “computing intelligence” is a lot more interesting.

The topic of customer interest is addressed at timestamp 59 minutes and zero seconds of the keynote.

By my count, 49 corporate logos were shown as launch partners.

At one point in the talk the CEO claims that “Blackwell will be Nvidia’s most successful product ever”.

—

And again, the product he’s talking about is a data centers packed with 56 racks…

That's not very convincing against Nvidia. Since they single handedly created the modern AI market through CUDA.

I don't think we'll be legally prohibited from buying it but there will be zero English docs (see Allwinner and such). Maybe if you're lucky you'll get an uncommented code dump with a forked years-old version of PyTorch.

From that exchange, clearly not. They do not have their stuff together. This is one area where AMD really should be employing quite a full team to ensure that major projects, including drivers, are able to support AMD cards without any friction. Whilst it's still a PITA to work.with AMD hardware in most AI frameworks Nvidia will continue to dominate.

In 50 years, I feel like GPGPU compute will be told as a Greek tragedy. Nvidia, pariah of Apple, would create a parallel compute ecosystem that was so powerful that even the combined effort of the industry couldn't topple it. At every corner where they could have thwarted them, Nvidia's competitors refused to up the ante or chided CUDA's efforts as silly and unnecessary. One crypto/AI craze later, everyone and their mother is berating their OEM for ignoring such basic functionality and refusing to collaborate on a competing alternative.

Frustrating is a good way to put it, but there's a certain causal satisfaction I feel from watching it all unfold. Of course everyone loses to CUDA when they refuse to sponsor an Open Source alternative. It's fascinating to me that hardware manufacturers would rather let CUDA dominate than establish a basic working relationship.

Sorry about that. We are just getting started, so the lowest priority right now is the website.

Additionally, due to the KYC requirements around these GPUs (due to US export controls), we really want to get to know our customers first.

Feel free to ping me on email and happy to get on a call and talk more.

Great question! I'm going to answer it the only way I know how... with a bit of a story of the history of things. Sorry if this bores you.

The problem I realized over a year ago was that nobody had hourly rental access to high end AMD GPUs. In addition, access to high end Nvidia was equally difficult. I signed up for a CoreWeave account, put in my credit card and was told a few weeks later that my account was not approved.

In effect, the only way to get access to super high end compute, was to be involved in HPC and that requires connections. At the time, we also didn't even know if AMD was going to seriously adopt AI as a strategy.

My view was that there were actually two problems, lack of general access and that everyone was putting all their eggs into a single basket. Mostly because of that lack of access, and because AMD was lacking a great developer flywheel story.

I spent August to December building a business plan, closing funding, forming the business, hiring my co-founder full time, securing data center space, securing direct relationships with vendors, and designing the system we were going to deploy. There are a million other little details in there, but this is long enough as it is.

Oct/Nov of last year rolls around and suddenly AMD has changed their tune. Lisa Su doubles down. Dec 6th, MI300x rolls out. We made our first PoC order in January, received it in March. It just goes to show how cutting edge and how long all of this takes. 3 more small (not hyperscaler) businesses sprung up during that time, all offering effectively the same product. We went to the data center, deployed our PoC and about 2 weeks later, we had our first customer onboarded. I call all of that validation, and was able to secure further funding based on it.

To answer your question, I'm not sure that I need a specific USP. The demand for compute isn't going down. If I have a product that people want, and I can offer them ethical, honest, truthful, great service around that product. All based on decades of experience. Can't that be enough? Myself and my investors believe so.