Building a Silicon Brain | Hacker News Reader

Building a Silicon Brain

the-scientist.com

73 pointsyigitdemirag17 comments

Among the problems you get with the neuromorophic chips is that they are choosing which hypothetical neural algorithm the chip should use and which silicon implementation should be made of that algorithm.

Current AI is continuously modifying it's algorithms. Personally, I'd want to see parallel chips with a programming interface at least as "general purpose" as the GPU.

One might even boldly say something like "the only way this deep learning explosion happened is through the consumer market making powerful parallel chips ubiquitous". Or more cynically, "if chip maker had been looking at only AI applications, they would have strangled the deep learning explosion by their tendency to demand $10K+ per installation since they look at the well-healed corporation that easily afford these and thus ignore the graduate students who make the conceptual progress in their basements." Even Nvidia is eager to divide their offerings between $500 game boards and $5000+ deep learning boards. One assumes a neuromorophic chip would be way up this "high end".

Most neuromorphic chips are general purpose in that you can tune the parameters of the neuron model. You can run a variety of learning algorithms by tuning these parameters. The only thing that is fixed is the model of a neuron itself.

To have hardware that allowed any neuron model would boil down to having an array of MAC units, or even more generally, a DSP. At that point you’ve lost sight of your original goal with a neuromorphic chip — to build an energy-efficient, scalable neural emulator.

With respect to the academic research aspect — graduate labs can get their hands on neuromorphic hardware. GPUs weren’t made ubiquitous to process scientific algorithms. It was the researchers who had the ingenuity to use GPUs for general purpose compute. Access to a powerful GPU then was as straightforward as access to a neuromorphic chip is now. The issue is when you try to scale the problem you are solving. In this vein, a GPU cluster of that size is not currently available to academics. That’s why you see a bias towards deep learning research in industry. In other words, it doesn’t make sense to compare a server of neuromorphic chips (which is what is required to simulate realistic neural behavior) to a single GPU. Compare it to servers of GPUs like Google’s or FB’s. You’ll see that academics have poor resources in both cases and the research keeps moving on.

And why not the GPUs themselves? Which inherent limitations do they have? Aren't they essentially general purpose parallel devices?

No, GPUs are quite different than CPUs with more cores. They're optimized for Singl-Instruction-Multiple-Data algorithms, in which the same operation is done at the same time to many different inputs (think image or video processing). They do very poorly in cases with complex branching logic.

I know, but isn't that what the OP was asking for? Isn't that an inherent tradeoff? Or: isn't that what Neuromorphic hardware and deep learning are about (ie. composing many relatively simple functions without complex branching/logic)?

SpiNNaker is very open-ended: it consists of a huge number of ARM chips connected by a custom network fabric, and scales extremely well.

Transmitting analog voltages in neuromorphic circuits reminds me of a cool experiment I heard about a while ago. A team used an evolutionary algorithm to adapt an FPGA to recognize sounds. When they got the behavior they wanted, they tried copying the circuit to another FPGA and found that it didn't work because the algorithm had evolved dependencies on specific quirks in the original physical FPGA. I can't figure out how to link to the PDF directly, but the original research is probably "An Evolved Circuit, Intrinsic in Silicon, Entwined With Physics" by Adrian Thompson, which tells the story far better than I have.

It's neat to think that our own brains might have adapted wiring that works specifically for the actual molecules that make up our bodies. It's uniqueness on a different level.

Lovely article about the paper, one of my favorite reads:

https://www.damninteresting.com/on-the-origin-of-circuits/

Thx. it's great

On this topic, more specifically of whole-brain emulation, a very interesting report I found a few months ago: "Whole Brain Emulation: A Roadmap". It's one of a very rare breed: an evaluation of a future technology that does not enter into wild speculation or wishful thinking, but remains critical and objective as far as possible.

[1]: https://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf

It's unfortunately quite old, predating the DL revolution. I think it would be written much differently now: emphasizing more the possibility of brain-like architectures (perhaps using real brain data as rich 'supervision': https://www.reddit.com/r/reinforcementlearning/comments/9pwy... ) rather than direct brain emulation, and more skeptical about the extent to which deep accurate simulation down to the molecular level is really necessary for human-level intelligence.

I've suggested repeatedly to FHI & Sandberg that an update of the Roadmap would be a very good project, but given how little progress there has been on true brain emulation since then (especially compared to DL), I'm not surprised that nothing's happened.

Another interesting kind of hardware they have at INI (Giacomo Indiveri's institute) are the dynamic vision sensors (initially developed by Lichtsteiner, Posch, Delbruck, Berner, Kramer).

Those are cameras modeled after the human retina, and they overcome the processing and frame rate bottleneck of traditional cameras by using an event based architecture (event stream of pixel brightness changes -> no data if nothing changes). This allows microsecond latency in machine vision systems, without the high data rates you get with frame based cameras.

* Overview: http://siliconretina.ini.uzh.ch/wiki/index.php

* Company selling them: https://inivation.com/dvs/

* Another company selling them split off by one of the PHDs: https://www.prophesee.ai/

* Demo video: https://youtu.be/jnzPuDUsP4w?t=14

This neuromorphic stuff is super cool. I wish I could just buy some! I'm the author of spikeflow (github.com/colinator/spikeflow). I'd love to support neuromorphic hardware.

For the moment, biological neural networks (brains) are, I believe, many orders of magnitude more powerful, and especially more efficient than, current computers. But it's hard to say, because it's so hard to tease out the important parts of what the brain does, because it does so much. There are so many types of neurons, so many types of connectivity, so many interacting neurotransmitters, so many support cells that affect computation... almost like the brain is the result of a billion years of hack-upon-hack evolution...

This a very cool repository indeed! Have you compared the speed/accuracy of spikeflow with other SNN libraries like Brian2 on decent GPUs (or even TPUs)?

Thanks! No, not quite there yet. Most other libraries are much further along - mine is really an 'exercise'...

agreed, very cool repo. Are you looking for help? I have some hours of my life to devote for some cool ML stuff.

Yeah, totally! You can find me email from my github profile.

Reply on news.ycombinator.com