Comment by joe_the_user | Hacker News Reader

Comment by joe_the_user | Hacker News Reader

Parent Full thread

Among the problems you get with the neuromorophic chips is that they are choosing which hypothetical neural algorithm the chip should use and which silicon implementation should be made of that algorithm.

Current AI is continuously modifying it's algorithms. Personally, I'd want to see parallel chips with a programming interface at least as "general purpose" as the GPU.

One might even boldly say something like "the only way this deep learning explosion happened is through the consumer market making powerful parallel chips ubiquitous". Or more cynically, "if chip maker had been looking at only AI applications, they would have strangled the deep learning explosion by their tendency to demand $10K+ per installation since they look at the well-healed corporation that easily afford these and thus ignore the graduate students who make the conceptual progress in their basements." Even Nvidia is eager to divide their offerings between $500 game boards and $5000+ deep learning boards. One assumes a neuromorophic chip would be way up this "high end".

Most neuromorphic chips are general purpose in that you can tune the parameters of the neuron model. You can run a variety of learning algorithms by tuning these parameters. The only thing that is fixed is the model of a neuron itself.

To have hardware that allowed any neuron model would boil down to having an array of MAC units, or even more generally, a DSP. At that point you’ve lost sight of your original goal with a neuromorphic chip — to build an energy-efficient, scalable neural emulator.

With respect to the academic research aspect — graduate labs can get their hands on neuromorphic hardware. GPUs weren’t made ubiquitous to process scientific algorithms. It was the researchers who had the ingenuity to use GPUs for general purpose compute. Access to a powerful GPU then was as straightforward as access to a neuromorphic chip is now. The issue is when you try to scale the problem you are solving. In this vein, a GPU cluster of that size is not currently available to academics. That’s why you see a bias towards deep learning research in industry. In other words, it doesn’t make sense to compare a server of neuromorphic chips (which is what is required to simulate realistic neural behavior) to a single GPU. Compare it to servers of GPUs like Google’s or FB’s. You’ll see that academics have poor resources in both cases and the research keeps moving on.

And why not the GPUs themselves? Which inherent limitations do they have? Aren't they essentially general purpose parallel devices?

No, GPUs are quite different than CPUs with more cores. They're optimized for Singl-Instruction-Multiple-Data algorithms, in which the same operation is done at the same time to many different inputs (think image or video processing). They do very poorly in cases with complex branching logic.

I know, but isn't that what the OP was asking for? Isn't that an inherent tradeoff? Or: isn't that what Neuromorphic hardware and deep learning are about (ie. composing many relatively simple functions without complex branching/logic)?

SpiNNaker is very open-ended: it consists of a huge number of ARM chips connected by a custom network fabric, and scales extremely well.

Reply on news.ycombinator.com