Hardware for Deep Learning. Part 4: ASIC | Hacker News Reader

Hardware for Deep Learning. Part 4: ASIC

blog.inten.to

90 pointslelf8 comments

In 2018 Pete Bannon showed me the first prototype of Tesla's deep learning ASIC (for training). It was sitting out on a desk, attached to a big heat-sink, near Elon Musk's empty chair in Tesla's Palo Alto office

They were having trouble writing a compiler for their custom hardware. So I asked, why not just use a GPU, and use Nvidia's libraries and compilers? Bannon became angry and defensive, and insisted that Tesla couldn't become dependent on Nvidia

So that's another reason to build your own hardware

The math for ASICs works out if you’re going to mint hundreds of thousands of chips. Tesla is building 500K cars per year, so they definitely cross the threshold for economic viability. The NRE costs for the best TSMC node is easily >$10M, but that is peanuts for Tesla against 500,000 units.

Source: I once spent a year analyzing custom ASICs for Bitcoin mining.

But the parent was talking about training. I don't think tesla is training machine learning models in their cars.

Still it's $20 per car for the cost of training hardware, no matter how many units of this hardware is required, within reason.

This is a rather acceptable cost. And they may potentially have an upper hand in development of such ASICs, in-house tools for them, etc.

Of course there is a risk that the results won't be impressive. Likely it's a calculated risk.

Do you think it was the right choice? Like did the custom hardware save money and make them more competitive? They announced that their board can do 144 TFlops but TFlops is such a bad gauge on performance.

> Cerebras hasn’t released MLPerf results or any other independently verifiable apples-to-apples comparisons. Instead, the company prefers to let customers try out the CS-1 using their own neural networks and data.

I suspect that despite Cerebras being a massive technical achievement (wafer-scale computing!) it performs worse than standard GPUs, which is why they won't release benchmarks on standard models (e.g. resnet/transformers/etc)

Looking at the TPU architecture was fascinating for an old-timer like me. I looked into implementing backpropagation on a Connection Machine CM2 back in the late 1980s. I got to the point of implementing the training algorithm in C* (an SIMD variant of C). However, I ultimately decided in favor of using a shared memory multi-processor. The main problem was that time on my unversity's CM2 was very hard to come by.

Is there any particular options for a hobbyists?

Reply on news.ycombinator.com