The Challenges of Building Inferencing Chips

semiengineering.com

52 points·rajnathani··5 comments

The paper is not very clear about what they mean with inferencing. It seems they just mean calculating a prediction for new data using a trained machine learning model. Readers outside the field, like me, are likely to confuse inferencing with inferring (using logic), but that is out of scope, here.

Edit: find the real explanation here: https://blogs.nvidia.com/blog/2016/08/22/difference-deep-lea...

Also not to be confused with probabilistic inference.

There are too many overloaded words in statistics and related fields.

So dot product machines or otherwise?

I've played around with some of the new 'Edge' accelerators (specifically Edge TPU and Intel NCS 2). The performance and energy efficiency of the Edge TPU especially is really amazing for the price. Does anyone know of any other edge devices worth evaluating at a similar price point (other than Nvidia Jetson)?

If your model can fit within 10MB, and is based on CNNs, then Gyrfalcon’s [0] ‘compute in memory’ inference chips can in theory provide 3-4x higher TOPS/watt compared to the above inference chips you’ve mentioned, with around 12.6 TOPS/watt for one of their latest chips [1]. Their chips already seem to be in some new devices [2] [3].

[0] https://www.gyrfalcontech.ai/

[1] https://www.eetimes.com/gyrfalcon-unveils-fourth-ai-accelera...

[2] https://www.zdnet.com/article/ai-comes-to-the-edge-with-soli...

[3] https://www.prnewswire.com/news-releases/lg-launched-phone-p...

Reply on news.ycombinator.com