Can anybody give an overview of AI/ML/NPU/TPU/etc chips and pointers to detailed technical papers/books/videos about them? All i am able to find are marketing/sales/general overviews which really don't explain anything.
Am looking for a technical deep dive.
If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vector or a matrix) simultaneously (whereas the CPU would operate mostly serially - one at a time).
The advantage is (in a nutshell) that you can get a significant speedup. How much depends on the problem and how big a chunk you can process simultaneously but it can be a significant factor.
There are disadvantages that people ignore in the current AI hype:
* The speedup in a one-off gain, the death of Moore's law is equally dead for "AI chips" and CPU's
* It is extremely specialized and fine-tuned software you need to develop and run and it only applies to the above linear algebra problems.
* In the past such specialized numerical algebra hardware was the domain of HPC (high performance computing). Many a supercomputer vendor went bankrupt in the past because the cost versus market size was not there.
I remember when NVIDIA didn’t have hardware for ReLU.
The fact of the matter on Moore’s law is that we’ve got transistors, but not TDP to burn and have for years. These stupid big L3 caches are just: “fuck it, I’ve got die to burn”.
This is an old story, things migrate in and out of the “CPU”, but the current outlook is that we’ll be targeting specialized hardware more rather than less for the foreseeable future.
It was pretty sad timeshare as I recall. The only machine(s?) available to a population of what? 20,000? And about 40 terminals total. You had to sign up for 15-minute measured timeslots.
I came from a state school that had over 1000 terminals on a dozen machines, all unlimited time to students. It was a big shock to find the star of Silicon Valley had such crappy student services.
Bit reversal is used in Fast Fourier Transforms. Its not entirely surprising to me that you'd have specialized hardware for that operation.
Ref: https://en.wikipedia.org/wiki/Bit-reversal_permutation#Appli...
Could you elaborate? ReLU is max(x,0) and CUDA had fmaxf since CUDA 1.0 (2007).
This is a fairly random press piece off Google but there are a ton of them. Hard to tell whether it’s the hardware, the blob, or the process node unless you work there.
https://developer.nvidia.com/blog/accelerating-relu-and-gelu...
A hypothetical relu instruction could probably get away with much less power and die soace?
I think there are some important question marks still unresolved that bear on how things will play out. E.g. how the training versus inference balance will land in terms of usage and economics.
Inference is inherently more "mass market". You need it locally without lags from moving data around. But inference is just numerical linear algebra. Ultimately augmenting the CPU to provide inference natively might be the optimal arrangement.
This does not seem to be true in reality. The H100 is about 2.5x 'better' for AI than the A100 (obviously depends exactly what you are doing), and they released about 2 years apart. That is roughly in line with Moore's law.
Probably we will see usage of large AI models in smaller devices the next years because there's another way to optimize: use more efficient representations of the model weights. I think about posits, a different floating point system where even 6 bits are perhaps usable. When models can switch to 6 bit posits from f16 (half floats), hardware can load more than three times larger models. We will see whether hardware for this will be mass-produced.
Technically, for AI, you need to accelerate numerical non-linear algebra calculations, which take the general form of matrix multiplication, but interpose non-linear functions at key points in the calculation.
That said, the GPUs in your M1 Mac are faster than the AMX units and any reasonably modern NVIDIA GPU will wipe the floor with the AMX units or Apple Silicon GPUs in raw compute. However, a lot of software does not use CUDA by default and for small problem sets AMX units or CPUs with just AVX can be faster because they don't incur the cost of data transfers from main memory to GPU memory and vice versa.
[1] Benchmarks:
https://github.com/danieldk/gemm-benchmark#example-results
https://explosion.ai/blog/metal-performance-shaders (scroll down a bit for AMX and MPS numbers)
Not for double, which is what R mostly uses IIRC.
Like compute eigenvalues/eigenvectors of large matrices, compute SVDs, solve large sparse systems of equations, etc?
[1] https://github.com/tensorflow/tensorflow/tree/8d9b35f442045b...
[2] https://github.com/pytorch/pytorch/blob/6e3e3dd477e0fb9768ee...
Could you please write what are some common day to day life applications of linear algebra in computing?
Computer graphics mostly because you can view the real world as 3d space and the screen as 2d space, and linear algebra gives you all the tools to manipulate something in 3d space and project it into 2d space. Neural networks because you can treat them as matrix multiplications.
Others have commented on computer graphics, but (as it turns out) the exact same algorithms apply for Collision Detection in 3D space. And since games already are manipulating graphics, you add on another set of Linear Algebra transforms that change the position/rotation/shear of those vertices. In a similar way, science (especially physics) use linear algebra to build simulations of all kinds of systems.
One surprising use is in Advertising (and other user preference aggregators). Turns out a preference acts like a magnitude of a one-dimensional vector. String N preference vectors together and you get an N-Dimension Vector that you can perform Linear Algebra operations upon. One common application is the Dot Product, which is a fancy way of taking two N-Dimension vectors and measuring how close those vectors point in the same direction in a [1, -1] range.
Yet another common place to find Linear Algebra is in computer science papers. Most of the time this is simply notation; a lot of common programming forms can be represented by MxN matrices. However some of those algorithms will use LA as a way of parsing and breaking down a problem. You will see this in compiler papers often, but its transferable to many other domains.
As a final and personal observation, I found that Linear Algebra helped me grasp Functional Programming. In both cases I am applying a transform to some input, and often stringing together many such transforms. Also in both cases, the transformations are sensitive to their order, and a bad ordering produces nonsense just like garbage data.
That's cosine similarity, or normalised dot product. The dot product can take any value when the vectors are not unit norm.
It's a matrix multiplication (https://en.wikipedia.org/wiki/Matrix_multiplication) accelerator chip. Matrix multiplication is important for a few reasons. 1) it's the slow part of a lot of AI algorithms, 2) it's a 'high intensity' algorithm (naively n^3 computation vs. n^2 data), 3) it's easily parallelizable, and 4) it's conceptually and computationally simple.
The situation is kind of analogous to FPUs (floating point math co-processor units) when they were first introduced before they were integrated into computers.
There's more to it than this, but that's the basic idea.
Some are just hardware pipeline to compute 2d/3d convolutions + activation function (Rockchip RK3588 has one like that). You give it the memory address + dimensions of the input matrix, the memory address + dimensions of the weights, memory address + dimensions of the output, which activation function you want (there are only like 4 supported), then you tell it RUN, you wait for a bit, and you have the result at the output memory address.
(I took Rockchip example to show that even in one microcosm it can change a lot)
And then you can imagine any architecture in-between.
AFAIK they all work with some RAM as input and some RAM as output, but some may have its own RAM, some may share RAM with the system, some might have mixed usage (RK3588 has some SRAM, and when you ask it to compute the convolution, you can tell it either to write to SRAM or system RAM)
It's possible that there are some components that are border line between ISP (Image Signal Porcessing) and a NPU, where the input is the direct camera stream, but my guess is that they do some very small processing on the camera stream, then dump it to RAM, then do all the heavy work from RAM to RAM. I think that Pixel 4-5 had something like that.
Is Rockchip's stuff "good", as NPUs go? I'm thinking of buying another 3588 SoC for my robotics hobby - you seem like you'd know if that's a decent idea or not.
Anyway, I really don't recommend it, unless you're making your own model, you know before-hand what's supported and what isn't, and your input is fixed resolution (which is a pretty fair usage in an embedded system) (fixed = doesn't change at every frame. handling hotplug from one webcam with a resolution to another with another resolution is fine)
I think looking at the examples give you a reasonable show of what it can do: https://github.com/rockchip-linux/rknn-toolkit/tree/master/e... It's mobilenet, yolov3, resnet50. There aren't more examples because they didn't had more examples. There aren't more examples because that's pretty much all you can reasonably run.
As far as I can tell, modern image models using transformer/vit won't be runnable on it. (it acts enough as a coprocessor that it's possible to do some parts in CPU some parts in NPU - and Rockchip framework handles that -, so maybe it's somehow possible)
(Note: I say this as a huge Rockchip lover, their mainline support is top-notch, they make very durable product (their 2015's RK3288 is still far from obsolete), and I bought a RK3588 SBC to play with a NPU accelerator (whose full specification is publicly available btw), in the hope to have a self-hosted LLM voice assistant)
Curious if anyone can weigh in on their SW usability. A quick search for their user level tools showed examples/documentation in Chinese(?)
I do know they have their own special sauce to talk to the NPU. I was discouraged from making the effort myself because their special sauce to talk to the VPU has barely any ffmpeg support, it p much only uses gstreamer, and I'm neither a masochist nor French so that's a non-starter.
By VPU are you talking about stuff like ISP, video encoder/decoder, or something else?
Among embedded processors I’ve seen touting vision acceleration, gstreamer support is fairly widespread. I bit the bullet to learn it because my role requires it. Maybe it’s Stockholm Syndrome talking, but I’ve somehow grown to like gstreamer. The learning curve was awkward. I struggled with documentation and learned more by analyzing some examples and trial-and-error.
Oh no. The prospect of gstreamer being the only way... Oh no.
Maybe there's a zsh plugin or smth that autocompletes sane defaults? AAAAAAAAHHHHHH there surely isn't, anyone merciful enough to make one would just use ffmpeg instead...
GPUs were optimised to draw, so were able to do dozens of these at a go. So these can be used for AI/ML in both gradient descent and inference (forward passes). Because you can do many at a go, in parallel, they speed things up dramatically. Geoff Hinton experimented with GPUs exploiting their ability to do this, but they aren't actually optimised to do that. It just turned out that it is the best way available to do it at the time, and still currently.
AI chips, are optimised to do either inference or gradient descent. They are not good at drawing like GPUs are. They are optimised for machine learning and joining other AI chips together so you can have massive networks of chips that can parallel compute.
One other class of chips that has not yet shown up are ASICs that mimic the transformers architectures for even more speed - though it changes too much at the moment for it to be useful.
Also because of the mechanics of scale manufacturing: GPUs are currently cheaper per flop of compute as the aggregate of scale is shared with graphical uses. Though with time if there is enough scale AI chips should end up cheaper
I read the first book in this list about 10 years ago, and though it’s pretty old the concepts are solid.
Even “manual” CUDA optimizations deal more with concurrency and data residency than parallelism and even those are usually limited to following the compute guide for your specific hardware and feature set and the driver does the majority of the heavy lifting.
https://online.stanford.edu/courses/cs217-hardware-accelerat...
Course website with lecture notes: https://cs217.stanford.edu/
Reading list: https://cs217.stanford.edu/readings
Here's a decent overview from the horse's mouth. https://cloud.google.com/blog/products/ai-machine-learning/a...
It's called a systolic array because the data moves through it in waves similar to what an engineer imagines the heart looks like :)
- GPU (Graphics Processing Unit)
- TPU (Tensor Processing Unit): ASIC designed for TensorFlow
- IPU (Intelligence Processing Unit): Graphcore
- HPU (Habana Processing Unit): Intel Habana Labs' Gaudi and Goya AI
- NPU (Neural Processing Unit): Huawei, Samsung, Microsoft Brainwave
- VPU (Vision Processing Unit): Intel Movidius
- DPU (Data Processing Unit): NVIDIA data center infrastructure processing unit
- Amazon's Inferentia: Amazon's accelerator chip focused on low cost
- Cerebras Wafer Scale Engine (WSE)
- SambaNova Systems DataScale
- Groq Tensor Streaming Processor (TSP)
If you don't mind, could you add links to authoritative wiki pages/whitepapers/articles for each of the above? I think it will give us a good starting point to start our study/research from.
OP asked about chips for AI.
System Architecture of these chips with detailed Functional Units and how they are used by the AI algorithm Instruction/Data streams.
Some random examples of video titles from the last 6 months of the channel:
* A Deep Dive into IBM's New Machine Learning Chip
* Does my PC actually use Machine Learning?
* Intel's Next-Gen 2023 Max CPU and Max GPU
* A Deep Dive into Avant, the new chip from Lattice Semiconductor (White Paper Video)
* The AI Hardware Show 2023, Episode 1: TPU, A100, AIU, BR100, MI250X
I think the podcasters background is actually in HPC (High Performance Computing), i.e. super computers. But that overlaps just enough with AI hardware that he saw an opportunity to capitalize on the new AI hype.
It's worth noting there are very widely deployed chips primarily built for inference (running the network) especially on mobile phones.
Depending on the device and manufacturer sometimes this is implemented as part of the CPU itself, but functionally it's the same idea.
The Apple Neural Engine is a good example of this. This is separate to the GPU which is also on the CPU.
Further information is here: https://machinelearning.apple.com/research/neural-engine-tra...
The Google Tensor CPU used in the pixel has a similar coprocessor called the EdgeTPU.
At the end of the day, it's matrix multiplication acceleration mostly, and then IO optimization. Literally most of the optimization has nothing to do with compute. We can compute faster than we can ingest.
Right. AFAIK we already were doing SIMD, Vector Processing, VLIW etc. to speed up parallel processing for numerical calculations in AI/ML. What then is the reason for the explosion of these different categories of chips? Are they just glorified ASICs designed for their specific domains or FPGAs programmed accordingly? If so what is their architecture i.e. what are their functional units and how do they work together?
GPUs have a high speed output in the form of an HDMI link. However, there is no high speed input. Reads/writes to/from the GPU are slow. The Cerebras wafer chip for example has 8-16 FPGA driven IO chips that directly read from TCP/IP onto the chip and off again in parallel. Each FPGA connects to its own ethernet port. So you can get the data on/off the chip as fast as possible. That's it really.
As for the processing engines. They're usually just standard cores with a high speed interconnect and maybe some matrix multiplication optimizations. Some, like Groq, have a high speed fabric with specialized processors at various locations.
https://medium.com/@adi.fu7/ai-accelerators-part-i-intro-822...
Please do submit this as a top-level HN submission so that it gets greater visibility.
It's the founder of a new AI chip company and they talk a bit on the differences
Incidentally there used to be a proper "AI" chip. The original perceptron was intended to be implemented in hardware. But general purpose chips evolved much faster.
If you never have to change the network - for instance to do image segmentation or object recognition - then you can’t get any more efficient than a custom silicon design that bakes in the weights as transistors.
Of course, this is all pretty expensive still. If your models are small enough you can get away with even older GPUs with less VRAM like a GTX 1080 Ti. And then of course there's services like Google Collab and vast.ai where you can rent a TPU or GPU in the cloud.
I'd check out Tim Dettmers' guide for buying GPUs: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...
https://www.tomsguide.com/news/google-pixel-7s-most-critical...
https://www.macobserver.com/tips/deep-dive/what-is-apple-neu...
So dedicated hardware to do math stuff
When you're multiplying two large matrices together (or other similar operations) there are thousands of individual multiply operations that need to be performed, and they can be done in parallel since these are all independent (one result doesn't depend on the other).
So, to train/run these ML/AI models as fast as possible requires the ability to perform massive numbers of floating point operations in parallel, but a desktop CPU only has a limited capacity to do that, since they are designed as general purpose devices, not just for math. A modern CPU has multiple "cores" (individual processors than can run in parallel), but only a small number ~10, and not all of these can do floating point since it has specialized FPU units to do that, typically less in number than the number of cores.
This is where GPU/TPU/etc "AI/ML" chips come in, and what makes them special. They are designed specifically for this job - to do massive numbers of floating point multiplications in parallel. A GPU of course can run games too, but it turns out the requirements for real-time graphics are very similar - a massive amount of parallelism. In contrast to the CPUs ~10 cores, GPUs have thousands of cores (e.g. NVIDIA GTX 4070 has 5,888) running in parallel, and these are all floating-point capable. This results in the ability to do huge numbers of floating point operations per second (FLOPS), e.g. the GTX 4070 can do 30 TFLOPS (Tera-FLOPS) - i.e. 30,000,000,000,000 floating point multiplications per second !!
This brings us to the second specialization of these GPU/TPU chips - since they can do these ridiculous number of FLOPS, they need to be fed data at an equally ridiculous rate to keep them busy, so they need massive memory bandwidth - way more than the CPU needs to be kept busy. The normal RAM in a desktop computer is too slow for this, and is in any case in the wrong place - on the motherboard, where it can only be accessed across the PCI bus which is again way too slow to keep up. GPU's solve this memory speed problem by having a specially designed memory architecture and lots of very fast RAM co-located very close to the GPU chip. For example, that GTX 4070 has 12GB of RAM and can move data from it into its processing cores at a speed (memory bandwidth) of 1TB/sec !!
The exact designs of the various chips differ a bit (and a lot is proprietary), but they are all designed to provided these two capabilities - massive floating point parallelism, and massive memory bandwidth to feed it.
If you want to get into this in detail, best place to start would be to look into low level CUDA programming for NVIDIAs cards. CUDA is the lowest level API that NVIDIA provide to program their GPUs.
1 - It's RTX 4070, not GTX 4070
2 - the 30 TFLOPS you mention are at the very top when overclocked, they go for 22 normally.
3 - Also those are single precision TFLOPS, as in 32 bit. What really matter nowadays is double precision. And in double precision a 4070 is 0.35 TFLOPS (or 350 GFLOPS). 2 orders of magnitude lower, still impressive though
Yes, the exact numbers are going to vary, but just giving a data point to indicate the magnitude of the numbers. If you want to quibble there's CPU SIMD too.
If your application calls for generality - like a gaming card's need to run custom shaders, or an ML model's need to run custom compute kernels, then an ASIC won't help you. These applications still need a general purpose processor, just one that provides huge parallelism.
It seems you may be thinking that all an ML chip does is matrix multiplication, and so a specialized ASIC would make sense, but that's not the case - an ML chip needs to run the entire model - think of it as a PyTorch accelerator, not a matmul accelerator.
Finally, the market for consumer (vs data center) ML cards is tiny relative to the gaming market, and these chips/cards are expensive to develop. Unless this changes, it doesn't make sense for companies like NVIDA to develop ML-only cards when with minimal effort they can leverage their data center designs and build dual-use GPU/compute consumer cards.