'Mind-blowing' IBM chip speeds up AI | Hacker News Reader

'Mind-blowing' IBM chip speeds up AI

nature.com

41 pointsbookofjoe36 comments

From Ars[0] (emphasis mine):

"It's worth clarifying a few things early here. First, NorthPole does nothing to help the energy demand in training a neural network; it's purely designed for execution. Second, it is not a general AI processor; it's specifically designed for inference-focused neural networks. As noted above, inferences include things like figuring out the contents of an image or audio clip so they have a large range of uses, but this chip won't do you any good if your needs include running a large language model."

[0] https://arstechnica.com/science/2023/10/ibm-has-made-a-new-h...

So having memory on-die is revolutionary? Better tell Apple who has been doing it for years.

And Tesla’s Dojo is already running processors and memory together at wafer scale, as are a number of startups.

I love it when uninformed writers just regurgitate the press release rather than do research to give context.

I do generally agree with your point here that this isn't very revolutionary. However it's worth pointing out that Apple chips have lots of on-package memory but relatively little on-die and they are quite different from this IBM chip because of that.

I know basically zero about chip fabrication but I remember reading somewhere a while (15÷ years) ago that processor-in-memory was always a desirable design objective for obvious reasons, but that there are fundamental differences in the process for fabbing memory versus logic (different regions of Si doping not possible in same wafer, something like that? See this is where I should stay out of these discussions) that haven't been resolved, so the next best thing is on package pairing.

Roughly speaking, CPU wants smaller space and RAM wants bigger space. At a high-level take:

  * CPU design is the most expensive space due to it having the greatest quality and capability requirements.  RAM is mostly just a very, very large repetitive structure, so more space better.
  * A CPU fault can be potentially corrected by microcode changes to route around the damage (ie part binning). RAM cannot generally take faults.
  * DRAM is simpler to make, "just" a capacitor, but capacitance leaks over time; which, means generating resistive heat in an area that we want as little heat as possible.  You could use SRAM (two transistors) but now you have substantially more complex part to fail.
  * DRAM quality requirements are much less stringent if you make just bigger cells.

>Neural inference at the frontier of energy, space, and time

https://www.science.org/doi/10.1126/science.adh1174

Thanks, this the underlying processor, IBM's "NorthPole", architecture paper.

Threadripper 7980X has 320MB of cache - that's 50% more than this mind blowing chip. We'll have to think of some new superlatives.

The 7995WX has 480mb, >100% more.

And going the other way, the consumer 7900x3d has 57% as much L3 cache (128MB)...

And only 64 cores, versus 256 for the IBM chip. The "mind-blowing" part is about its efficiency, not about its total memory or core count.

Is there some optimal memory amount per core where going above or below is less efficent?

No clue, this is about energy efficiency though. Perhaps I should have said that, but it was in the article so I figured people would understand.

It's also pretty clear this is a prototype, not a final product that can never be improved in any possible way.

The article is saying that the efficiency comes from the use of on-die memory, avoiding the von-neuman bottleneck

And with Sapphire Rapids having 64GB of HBM2e, I think even TR's is quite miniscule.

>But even NorthPole’s 224 megabytes of RAM are not enough for large language models, such as those used by the chatbot ChatGPT, which take up several thousand megabytes of data even in their most stripped-down versions.

Is that right?

Yeah. 224 MB is nothing.

Small LLMs that have been quantized to optimize for space still sit at 4-9 gigs

That's not true, there are some very pruned (and relatively dumb) LLMs which go under a gig.

What’s the point of a state of the art AI chip that can’t run large models? It seems problematic to say the least!

It can run small models very fast.

It is a new design, not a direct competitor for nVidia’s flagship. And there are lots more applications for AI than LLMs.

Realtime, low power, isn't really possible today. Think anything motion or reaction related.

Some of us still work on computer vision. 224MiB is fairly massive for convolutional neural network.

Small LLMs or large small language models???

I thought the article was mixing up MB and GB at first. Is that the size of the LLM with no training data? Is there any value in that?

These are the trained weights and biases, the training data is unknown in size but could be terabytes… I’ve no idea how to even guess at the size of the training data but that doesn’t all need to in ram at the same time.

Once the model is trained, it doesn't need to keep the training data around.

GPT-3 is about 175 billion parameters (though I have no idea how many bits per parameter OpenAI uses at inference-time), and is apparently trained on 45 TB of data[0]

[0] Caution: citation was first hit on google, YMMV — https://www.springboard.com/blog/data-science/machine-learni...

Presumably if you are training a robot to use a new/different tool you'll want the ability to train on site. If you buy an iHop restaurant the pancake robot in the kitchen ought to be able to be repurposed as a hamburger robot for your cheeseburger business. Omlette scrambling robots could be trained to mix small batches of cookie dough. Etc etc. Toyota is working on developing a framework for this already.

Some robot firms are integrating LLMs to make the robots more general: https://www.youtube.com/watch?v=Vq_DcZ_xc_E&t=2s

On-site training is… not really solved yet. Not efficiently, at any rate: any task can be trained with sufficient compute and/or examples, but probably more than most companies would care to bother with, and certainly more than we'd get onto one of the chips in the article.

That's not to diss the chips: As I understand it, one of the biggest issues is the power envelope of mobile units, which means making the computations more energy efficient is going to help massively, it's just that "training" and "inference" are currently very distinct tasks with very different hardware requirements.

(Also, I'm not sure if you mean those examples as illustrations or are serious about them: if you're serious, I suspect an old-fashioned robot arm bolted to the ground and following a pre-programmed path will probably cover your needs — GOFAI is great in restricted domains, the more modern AI models are more appropriate when the environment is more chaotic and less predictable, such as collaborating in a kitchen that also has humans or being asked on the fly to do a new recipe it's never encountered before).

I would expect LLM hardware to routinely support between 32 and 512GB memory in the Very Near Future. 1-4TB by the end of the decade. Custom hardware for GPT and LLM technology only started being developed in earnest in September 2022

At a glance, the supplementary materials from IBM's paper claim even less:

> NorthPole's core array includes 192 MB of flexible memory (768KB of unified memory per core). Assigning 2/3rd of this memory to parameters, such as weights and biases, provides 128MB for network storage.

https://www.science.org/doi/suppl/10.1126/science.adh1174/su...

My understanding is that this is a small energy-efficient chip for edge computing, like to stuff in an IoT device. Way too little memory to expect to run any recent language models, but could maybe do some basic object detection in a doorbell camera, for example.

The article's author seems to believe 224 megabytes is a huge amount of memory, and is a few orders of magnitude too low on the ChatGPT estimate too.

IBM press release https://research.ibm.com/blog/northpole-ibm-ai-chip

So it is like a TPU then.

Or the Apple M series processors...

Any possibility that it would hit the mass-market?

Does not sound any better than Graphcore's chips which had 900MB of SRAM and 1000 cores a couple of years ago.

The problem is that even that amount of memory is tiny by today's LLM standards.

Reply on news.ycombinator.com