Hacker News Reader

Comment by jimbokun | Hacker News Reader

In the past, specialized architectures tended to lose out to the performance gains of generic x86 chips. I'm thinking RISC vs. CISC, or Sun's attempt at a custom Java-optimized processor.

Is that no longer the case? ARM architectures seem to be beating x86 for low power, mobile devices. GPUs are being used for many easily parallelizable workloads.

Is an overall slow down in Moore's law making chips designed for specific tasks (like Deep Neural Nets), attractive again?

View on HN

Deep Neural Networks involve Convolution and Matrix Multiplication operations which have a well defined hard upper bound in terms of speed wrt number of cores, GPUs (~thousands of processors) are in thus indispensable achieving a faster performance. There are newer interesting approaches such as reducing multiplication operations which might improve inference/forward speed even further. http://arxiv.org/abs/1510.03009

I'm not sure CISC beat RISC, except for complicating the instruction set enough to keep costs high for competitors.

Underneath, Intel processors translate x86 to simpler RISC like microcode. They could make a more efficient chip without this translation component, and probably holds them back a bit in low power stuff.

It's more like x86 beat everything, which is also saying that externally CISC/RISC didn't matter so much. I like the section on RISC here: http://danluu.com/butler-lampson-1999/ Specifically part of the last paragraph: "It’s possible to nitpick RISC being a no by saying that modern processors translate x86 ops into RISC micro-ops internally, but if you listened to talk at the time, people thought that having a external RISC ISA would be so much lower overhead that RISC would win, which has clearly not happened. Moreover, modern chips also do micro-op fusion in order to fuse operations into decidedly un-RISC-y operations."

You can do fast array math on video cards and specialized FPGAs; neural nets require fast array math. This chip is optimizing some memory overhead involved in DL. Probably lots of such optimizations are possible in DL, since it is a fairly well defined problem.

Yes, it's still the case. Moore's Law is still alive (e.g. Tegra X1 vs K1). However, this paper is not about using specialized architectures, it's about compressing weight matrices and putting them in cache.

x86 beat everything largely due to momentum. Intel had the greatest ability to invest in process shrinking technology and the brute force to engineer their way out of architectural dead-ends (see NetBurst).

Now that process shrinking has become extremely difficult, everybody else is starting to catch up.

Reply on news.ycombinator.com