Hacker News Reader

Comment by sawwit | Hacker News Reader

How come they aim for GPU technology rather than for neuromorphic computing ASICs?

I was heavily into neuromorphic hardware ~5 years ago when the hype seemed to be at its peak. We funneled a load of time and money into it. It was working okay for what we were doing and everybody was generally happy.

Then the NVidia Tesla C2050 came out and CUDA reached a somewhat stable release. The neuromorphic plane has crashed into the proverbial mountain.

EDIT: Maybe I should elaborate - we accomplished the same task using the Tesla card, but it was about 100x faster and each card was $2500 and usable for other tasks. The general rule of thumb became: design a neuromorphic system, wait for the next Tesla chip, then simulate it in software.

Contrary to what you might read in puff pieces from IBM PR, all existing "neuromorphic" hardware is terrible. The neural nets that work don't look anything like the brain, and neural nets that look like the brain don't work.

People constructing neural nets that look like the brain are engaged in cargo cult science, because the truth is we have no idea yet how the brain works and attempting to imitate it without knowing that is doomed to failure.

I'm curious to hear your opinions on Numenta's work on emulating parts of the neocortex.

He just gave it to you. Their system is not really competitive in any metric people are measuring.

A good measure of any of the techniques are the results. We see record object recognition and speech recognition results. How many of these results are from Numenta or IBM neomorphic chips? Most successes have been on deep learning architectures over GPUs (and large datasets). While these fancy architectures may have their applications, they have nothing to do with recent advancements in last 3 years.

To be fair, the goal of those new computing architectures is not to advance algorithms. That at best is a side benefit.

Those algorithmic performance improvements seen in DNN have come from improved datasets and training. Numenta and IBM arent focusing on training AFAIK. Google's Quantum Annealing [0] is the only hardware I'm aware of focused on training, although there are rumors Nervana Systems may produce something [1]. I'm sure there are others; accelerating training of DNN isnt a particularly new idea.

The goal of these other computing architectures is typically to provide lower power, higher frequency/lower latency, or smaller form factor execution of trained models, but there is a question of how much value they can provide over more conventional chips to be worth the chip design costs.

However without these architectures becoming as mainstream as say a GPU, I think we will continue to see advances come from the typical everyday computer. The ML community seems to be much more democratic than others.

[0] http://googleresearch.blogspot.com/2015/12/when-can-quantum-...

[1] http://www.nervanasys.com/about/

Numenta benchmarks their algorithms, along with others, on real-time streaming data: http://numenta.com/numenta-anomaly-benchmark/

I agree that neuromorphic is often specious, but it's not fair to say that neuromorphic hardware fails if it can do non brain like computations more efficiently than conventional hardware.

The thesis may be misguided at times but building non-traditional hardware that provide significant efficiencies is a wonderful thing. Sometimes it takes a random vector to get off a local maximum.

For example, did you know Synaptics, the guys who developed the touch input you're probably using started as a crazy neuromorphic thing from Carver Mead? [0]

While most of the IBM press is garbage, they do seem to be making progress running conventional DNNs on their hardware, showing backprop in low precision, and suggesting progress on CNNs. [1,2]

[0] https://en.wikipedia.org/wiki/Carver_Mead#Touch

[1] https://papers.nips.cc/paper/5862-backpropagation-for-energy...

[2] http://p9.hostingprod.com/@modha.org/blog/2015/12/nips_2015_...

This is off topic, but Synaptics touchpad drivers have been the worst part of using a Windows laptop for years and years. Microsoft finally had to take matters into their own hands and write touchpad drivers themselves in Windows 10 because Synaptics just couldn't do it right.

Back on topic, that paper is pretty interesting, though MNIST is a toy problem. If they could run any of the ILSVRC winners then I would be impressed, but I can't help but think that if one wants to run CNNs one would be much better off designing a CNN chip to begin with rather than a wacky spiking architecture and trying to shoehorn a CNN into it. To my knowledge nobody has yet fabricated an ASIC specifically for CNNs so there's no way to fairly compare their approach to something like that.

Both good points. Yann LeCun (et al) have developed CNN FPGA designs where they project future ASIC performance, but I'm not aware that this has ever been taped out, nor any power numbers released:

http://yann.lecun.com/exdb/publis/pdf/farabet-iscas-10.pdf

IBM has been working on these chips from 2009 I think, before DNNs really became popular again and I bet they would change a few things if they were designing from scratch today - especially better support for CNNs. The cost of chip production for something like True North is very high (5.4B transistors!) so I suspect they are trying to squeeze as much out of it as they can :)

Which neuromorphic ASICs are you thinking of? GPUs are fast and they work now. Some "neuromorphic" chips, like IBM's True North, are just low-energy GPUs, so neuromorphic is in the eyes of the beholder...

This is not very accurate. The true North chip places memory next to compute elements on chip. GPUs have most memory off chip. The true North hardware is specialized to neural net stuff, and doesn't allow general floating point compute, have any sort of instruction set decoder, etc. That turns into big performance and power wins.

Just a guess, but they project the cost of large scale GPU manufacturing to decline faster than ASIC due to the shared applications of gaming/VR?

You would think AI would be bigger than VR.

Edit: going with GPUs makes sense now, but it doesn't indicate a big bet on Facebook's part. I don't think opensourcing a server cabinet is that interesting, but I'm a hardware engineer.

In 10 years I think more people will touch AI than VR but I don't know how that translates to investment.

Gaming is a proven market. Capital hates risk. (Real capital, not the fringe type we talk about here.)

AI might be bigger than VR, but the timeframes are messed up. VR is an engineering problem now; I don't think there are major field-shattering breakthroughs needed to make reasonably realistic, cheap VR within the next 5 years.

AI is nowhere close to strong AI or reasonable general-purpose intelligent programs that don't require loads of human tuning. It will take several breakthroughs.

Basically, you need hardware designers and a ton of money to develop ASIC's that will accelerate as good as current GPU's. I've seen designs along lines of SIMD's, DSP's, etc that might be a nice alternative in custom hardware. Still tons of money to get them to work. Doesn't even count developing compilers and such to utilize them properly.

Existing tools on GPU's are easier despite me wanting more work on FPGA's and ASIC's in this area. ;)

Reply on news.ycombinator.com