Nvidia is most valuable company

Modern ML is mainly made of a few parts:

1. large matrix multiplication

2. a few other things like element-wise operations, often exp() or other sources of nonlinearity.

3. data movement and reshaping

Accelerating all of those is easy. The issue is that ML is still not yet stabilized in terms of data flow between those steps. RAM bandwidths are not growing fast enough, which is why some latest ML developments trade off more compute for less RAM bandwidth. The field is developing fast. New proposals come out very often that change the way those building blocks get used.

GPUs are used because they are:

1. fast enough at mat-mul

2. flexible enough to reorder things and change what is sent where when and how.

Once ML field stabilizes a bit, and we're done with new papers coming out every month with new proposals for the ways to flow data through the steps, you can fully expect real ASICs for this. But if you build one now, either you need enough flexibility to change how it works on the fly (read: you're building a GPU) or your accelerator becomes useless once a new paper comes out (see: 4NX from imagination once the attention paper was published). Once the field has stabilizes a bit, it will happen. The reason bitcoin ASICs happened is because sha256 was stable and nobody had found new ways to calculate it much faster

"But what about TPUs?" you might scream! They are a large mat-mul block with a nearby general-purpose core with a large vector unit to try to do "the rest of the things". This is the alternative approach, with pros and cons compared to GPUs, but it is also basically a hedge against "new ML paper comes out between chip design being locked and chip taped out and in use". You could argue whether this is better or worse than GPUs. There are valid points on both sides