Hacker News Reader

Comment by pradn | Hacker News Reader

The AI boom is, in theory, a godsend for Intel and AMD. You can focus on creating good tensor computation hardware, without having to worry about getting gamers on board. No need for "Game ready drivers" or compatibility with complex legacy graphics APIs.

Of course, there's the elephant the room for general purpose tensor machines, which is CUDA - famously closely guarded Nvidia. But with the new wave of "above-CUDA" APIs like TensorFlow, PyTorch, and Keras, there's an opportunity to skip the CUDA layer altogether.

View on HN

The AI boom that lead to Nvidia's market position is still, 3 years on, almost entirely speculative. The only products that are currently driving value (i.e., that are providing consumer surplus, for which businesses and individuals are actually opening their checkbooks for) are paid chat-bots.

Nvidia ate the whole pie because they were in the right place at the right time with shovels to sell. Getting into it now, in house, would be a huge bet that the pot of gold will be there when they get to the end of the rainbow -- and they have a long way to go.

Intel actually has it even worse because they completely failed to bring ARC to market in time. The product came out (very) late, underdelivered, and permanently damaged the brand reputation. Yes, the drivers and thus the cards have gotten better, but who's talking about them now? Noone.

They say the opposite of love is indifference, and basically everything Intel has sold outside of CPUs and NICs have suffered market indifference. Actually, maybe even the NICs given everyone seems to prefer Realtek NICs over Intel NICs these days.

Nvidia and AMD have it way better because if the market doesn't love them, you bet the market will hate on them instead of be indifferent.

Realtek WiFi NICs are dogshit. Intel AX210 shine where the others are slow and unreliable.

It's off topic, but congratulations on the triple mixed metaphor. Not sure whether you use the shovel to dig up the pot of gold to buy the pie, or sell the shovels to bash through the pie crust and get to the rainbow....

The whole situation with Intel and AI is also baffling to me. They have an excellent product in this space - Gaudi2. Faster than A100 and very attractively priced. I’ve tried it, it works fine. Gaudi3 is also about to come out, and it’s twice as fast as Gaudi2. Yet nobody is buying. I get why you wouldn’t want this for training - it requires some minor code modifications and your Triton kernels are worthless. But for generative inference this is just what the doctor ordered.

> generative inference this is just what the doctor ordered.

Naive question, wouldn't you need a descent tool chain for inference as well ?

Assuming you mean “decent” toolchain, it’s actually pretty decent. Could it use some polish? Yes. But any decent ML engineer would be able to get a high performance server (or a batch job) running in a relatively short time. Or in almost no time at all if using a lot of the FOSS models. You just basically create a model in PyTorch and then hand it over to Gaudi stuff which patches it with optimized, Gaudi specific ops and converts things into an inference graph.

“Closeness to CUDA” is less important for inference because all the experimentation is already done by then, and if need be you could just implement the model using Gaudi ops to begin with, in a span of a few days including tuning and debugging.

Gaudi was made by an Israeli company Intel acquired in 2019 (not an internal project). Gaudi 3 has PCIe 4.0 (vs. H100 PCIe 5.0, so 2x the bandwidth). It's strange for Intel, of all vendors, to lag behind in PCIe. And Nvidia has SXM for larger models (5x bandwidth).

"N5, PCIe 4.0, and HBM2e. This chip was probably delayed two years." -wmf

PCIe doesn't matter - these accelerators talk to one another, and Gaudi is outstanding in this regard. HBM2e also doesn't matter if you run decent sized batches (which you should, for throughput). In fact, HBM2e, being far less supply constrained, might even be an advantage.

Besides that it does matter (e.g. loading/saving LLMs), it speaks how Gaudi is not even using Intel's latest technology. They are clearly not in the main production pipeline. So Gaudi is not going to save Intel.

But you don’t “load” models during inference. They are already on-device.

AI doesn't exist with the right amount of collected data, we're going to hit a wall very soon where we just get bad AI data constantly because we are throwing data to the wall and hoping it sticks. Businesses should not be investing in AI right yet maybe 2-3 more years.

The businesses that should are the startups and businesses who are helping define the model, like ChatGPT and Microsoft, but the adoption is still too early. If I was McDonald's or Wendy's and I wanted to use AI to help promote sales to customers I would need to be able to grab demographic data which may not be appropriate PII data.

All of the lawsuits happening right now for data collected without permission of the data provider is going to all change the landscape.

I think there is still a good business to be made taking the AI stuff we have right now and making it highly performant.

Stable diffusion models are awesome and super useful. I don't necessarily mean the simplistic text->image stuff (though it's clearly useful)-- but denoising, in painting, animation from stills, 3d from stills.... style changes, etc.

LLMs are at least somewhat useful.

We can radically improve image and video compression with performant enough AI.

Plenty of other applications of AI right now are already useful or would be useful if inference were made more efficient or more local (thus private). No more data needed.

You don't need some grand vision of an AI future to get a total addressable market for high performance AI that is in the same ballpark as high performance GPUs for non-AI usage.

But for every business the chase seems to fail in finding worth. I can already tell you from where I am working we are struggling because eventually data is too old to be good or too incomplete. We need at least 3-4 years of collecting and storing the information to make full use of AI.

In practice it's the other way around.

In practice AMD and Intel GPUs should be suitable for machine learning but the software story isn't good. I know I can buy a NVIDIA card and know it's a good investment because everything worth with CUDA and be training models in less than a day. If I went with some other brand I'd expect to put 6 man*months into figuring out the software story, and that's a lot more than the price difference in the cards.

I've been wondering about the soundness of Intel's software strategy in that OneAPI ("One" is a bad smell in marketing speak, the only premium product in my mind that has "One" in the name is "Purina ONE" pet food, the XBOX ONE is an astonishing own goal of a name because you'd never get your mom to understand that an XBOX ONE is better than an XBOX or XBOX 360) is based on OpenCL and the one thing I know about OpenCL is that I don't know anyone who likes coding for it. The idea that you could code for FPGA and GPU out of the same API also seems delusional because FPGA is (mostly) about latency and GPU is about throughput. That is, FPGA can do certain small operations insanely fast and avoid the overhead of 16-bit math when 13-bit math will do, GPU is all about doing large operations in bulk.

> If I went with some other brand I'd expect to put 6 man*months into figuring out the software story, and that's a lot more than the price difference in the cards.

Several companies are spending >$10B annually on AI compute. There are a lot more than 6 man-months of savings in it for them...

One firm saving $2B saves $2B, open source software that opens up the market for everyone is priceless.

A company like OpenAI that is mostly concerned about pace might see going with the #2 GPU vendor is risky, even if it has the possibility to save money.

OneAPI is an implementation of SYCL. Apart from being standardized by Khronos, SYCL and OpenCL don't have that much in common, except that they are often interoperable due to many SYCL implementations running on top of OpenCL. Intel specifically has two backends, one for OpenCL and one for the lower-level Level Zero API.

Saying OneAPI/SYCL is bad "because OpenCL" is the same as saying OpenCL is good "because CUDA" (since the NVIDIA implementation of OpenCL is called CUDA and uses the larger CUDA stack).

As for the FPGA/GPU code sharing, iirc FPGA vendors including Altera/Intel and Xilinx/AMD support OpenCL as well. The idea doesn't sound that crazy to me: GPU code also often uses small data types (fp16, tf32, int4), and both work with data parallelism. GPUs operate in parallel with SIMD vectors, but a compute accelerator FPGA achieves a similar thing by essentially pipelining the program. Effective pipelining requires parallelism too, since a serial FPGA program only has one stage active, similar to how a serial GPU program only has one ALU per compute unit active. As such both are mostly suitable for massively parallel programs.

>"One" is a bad smell in marketing speak.

Thank goodness other people recognize this too. Whenever I see “One” in branding I sense a corporation bereft of creativity and afraid of risk. Alternatively, it might also indicate that were too many executives involved and no single person had the authority to make a decision, so they picked the most bland and uninspiring option they could all agree on.

It’s so lame and milquetoast. However, there is one exception I will make, the radio station 101.1 “The One”. Here it feels appropriate and even a little endearing.

No there isn't. Let's take pytorch as an example.

CUDA is an API. OpenCL is an API. Pytorch is not.

Pytorch is very firmly GLUED to CUDA. It will probably NEVER support anything else beyond token inference on mobile devices. The only reason Pytorch supports AMD at all is because of AMD's "I can't beleive it's not CUDA" HIP translation layer.

OpenCL is a real cross-platform API and with 3.0 it's finally "good" and coincidentally, intel is....half-heartedly interested in it, except they're shooting themselves in the foot by trying to also cover useless CPUs for inference/training and spreading themselves too thin (OneAPI). Because all intel can think about are CPUs. Everything must drive sales of CPUs.

At this rate just about the only think that might save us from CUDA is rusticl. If a real, full-fat, high-quality openCL 3.0 driver sudently popped into existence on every GPU platform under the sun, maybe pytorch et al could finally be convinced to give a shit about an API other than CUDA.

Reply on news.ycombinator.com