Hacker News Reader

Comment by dragontamer | Hacker News Reader

> For example, the engineer says the custom ASIC does 144 TOps for 2 chips vs the NVidia drive Xavier - does 21 TOps. Okay, well yeah I expect your custom ASIC does have a nice performance advantage over the equivalent GPU. at 3.5x advantage probably seems reasonable. Cue Elon Musk:

Here's my issue with that. The on-chip SRAM is only 32MB, and the RAM is LPDDR4 rated at only 68GB/s.

Assuming a dot-product (multiply + add) over INT8 data, that's a limitation of 2-operations per 68GB/s (that the RAM moves at). Or 136 GIOPS (Giga-integer8 operations per second). You're limited by RAM, based on what I've seen in the presentation.

Unless their neural net is 32MB and fits entirely in on-chip SRAM. That seems unlikely to me...

View on HN

Nvidia already responded:

""Tesla was inaccurate in comparing its Full Self Driving computer at 144 TOPS of processing with Nvidia Drive Xavier at 21 TOPS," a spokesman said in an email. "The correct comparison would have been against Nvidia's full self-driving computer, Nvidia Drive AGX Pegasus, which delivers 320 TOPS for AI perception, localization and path planning." The statement also contends that "while Xavier delivers 30 TOPS of processing, Tesla erroneously stated that it delivers 21 TOPS. "

From what I understand, Pegasus consumes about 500Watts, compared to under 100 Watts for Tesla's FSD computer. Elon in particular emphasized the performance per watt (as it's always possible to cram more chips to increase performance if you ignore cost and power consumption).

The comparison made in the video: 500Watts for an hour consumes about 2-3 miles of range. In a city in slow traffic, going 12mph, that's a significant range reduction. So you might have a 10% improvement in range for the Tesla ASIC in low speed conditions.

Its incomparable though. The Pegasus has far more compute power.

Just because the Pegasus has 500W worst-case TDP doesn't mean that its average case would be 500W constant. If you scale back your code and idle parts of the GPU, you can drop the energy cost arbitrarily.

At least, that's how GPUs on desktops work. They only use a ton of power if you give them a ton of work. Write your code in an energy-efficient manner, and the 2080 Ti will drop down to 20W, or scale all the way up to 300W.

https://www.tomshardware.com/reviews/nvidia-geforce-rtx-2080...

Modern chips idle very well. With the right code, the Pegasus could be tuned to only use 100W (assuming good enough programmers). But Tesla's chip will NEVER be able to scale above 100W.

Pegasus's chip is more general purpose and doubtless has far more general purpose compute power. But that's irrelevant. What's relevant is Tesla's chip is optimized specifically for their NN pipeline whereas Pegasus is based off of general purpose GPU architecture, thus Tesla's chip achieves a better TOPS per Watt than Pegasus. And it'd be strange if it didn't.

"Tesla's chip will NEVER be able to scale above 100W" okay, based on what? Tesla has a higher performance chip in the pipeline right now, and they could've used more silicon to achieve more TOPS if they needed it.

EDIT: Pegasus has 500W at 320 TOPS. Tesla's has 72 Watts at 144TOPS. Thus Tesla's chip, because it's focuses specifically on Tesla's NN pipeline, is about to get almost 4 times to performance per watt of the Pegasus and is much cheaper. Tesla's NN chip wouldn't help your video game, and Tesla isn't intending to compete in all the markets Nvidia operates in.

> Pegasus has 500W at 320 TOPS. Tesla's has 72 Watts at 144TOPS.

Theoretical TOPS which can only ever execute within the 32MB SRAM that Tesla has created. Otherwise, Tesla's compute chip is stuck at 68 GBps LPDDR4 RAM. Pretty slow.

Pegasus uses HBM2 chips at 500GBps. Pegasus will be able to efficiently compute neural networks that are larger than 32MB in size.

Tesla is making big bets about this tiny 32MB SRAM. Bits and pieces of the CNN can fit in there, but almost certainly not the entire neural network.

You're right that this is a specialized chip. But even for NN / Deep Learning inference, it seems a bit underpowered to me from a RAM perspective

Specmanship doesn't matter. What matters is how fast it's able to execute on the task at hand and for what cost in terms of purchase price and energy.

Nvidia's offering can be really good, and so can Tesla's.

Convolutions have a much higher arithmetic intensity than dot products.

But in terms of CNNs, it's basically dot product is it not?Can you give an example of a CNN where it isn't?

He has a point. There's a bit amount of data-reuse in CNNs.

Hmm... it will depend on the CNN. There's probably a good neural network design that would take advantage of this architecture. IE: A well recycled convolutional layer that probably fits within the 32MB (load those weights once, use them across the whole picture).

So the whole NN doesn't necessarily have to fit inside of 32MB to be useful. But at least, large portions have to fit. (say, a 128x128 tile with 20 hidden-layers is only 300kB). Recycling that portion across the 1080 x 1920 input would be relatively cheap.

I herp-derped early on, there seem to be CNNs that would make good use of the architecture. Still, the memory bandwidth of that chip is very low, I'd expect GDDR6 or HBM2 to definitely be superior to the 68GBps LPDDR4 chip they put in there.

In a conv layer, weights are shared across many input features. E.g. assume a 1x1 conv layer with a 28x28x3 input. You only need to load 3 weights even though there are effectively 28x28=748 different dot products. In practice, the input and output activations can be stored on chip as well (except for the first layer) which means the ratio of operations to DRAM accesses can be incredibly high. For some real world examples, take a look at the classic Eyeriss paper[1] which finds a ratio of 345-285 for AlexNet and VGG-16 respectively. You can also check out the TPU paper[2] which places the ratio at >1000 for some unnamed CNNs. Compare that to your analysis which yields a ratio of 2.

[1] https://people.csail.mit.edu/emer/papers/2017.01.jssc.eyeris...

[2] https://arxiv.org/pdf/1704.04760.pdf

I think the key thing is that you assume a single dot-product per RAM load. In general you get higher compute performance by doing more than this. You load weights for a layer in the SRAM, then stream from RAM tiles of data that you can compute multiple operations on.

In the same way, to reach peak FLOPs on a GPU you better use the local/shared memory as much as possible.

Reply on news.ycombinator.com