New Hardware for Massive Neural Networks (1988) [pdf]
papers.nips.cc
papers.nips.cc
> Transient phenomena associated with forward biased silicon p + - n - n + struc- tures at 4.2K show remarkable similarities with biological neurons. The devices play a role similar to the two-terminal switching elements in Hodgkin-Huxley equivalent circuit diagrams. The devices provide simpler and more realistic neuron emulation than transistors or op-amps. They have such low power and current requirements that they could be used in massive neural networks. Some observed properties of simple circuits containing the devices include action potentials, refractory periods, threshold behavior, excitation, inhibition, summation over synaptic inputs, synaptic weights, temporal integration, memory, network connectivity modification based on experience, pacemaker activity, firing thresholds, coupling to sensors with graded sig- nal outputs and the dependence of firing rate on input current. Transfer functions for simple artificial neurons with spiketrain inputs and spiketrain outputs have been measured and correlated with input coupling.
The brain as far as I understand does so much with large, slow elements (neurons) by having them fill a volume, be sparsely activated (i.e. mostly a huge memory), and other advanced communication methods (temporal pulse position modulation/frequency from spiking? neurotransmitters?).
Current ML is more densely activated, high-frequency networks. I'm not sure we could revert to the brain-like architecture unless we could get the cost of sillicon manufacturing several orders of magnitude down, enough that we could just fabricate a large block of stacked complex elements. A large part of the philosophy of nodes would need to be reworked (much lower frequency, lower leakeage, lower power consumption), as processes are optimized for >100MHz freqs; just so internal memory elements would keep at acceptable temperatures. Currently you could fit about 2000 GPUs in a 10cm^3 space (assuming 1mm die thickness), which would cost about $1.5M usd. And couldn't do much, because it would quickly overheat on reasonable loads, and because I don't think we have the technology to interconnect it all.
The variation in neurotransmitters allows for different sorts of activation, typically with different physical parameters (size of activation, time over which it decays) and multiple ways they interact with the other neurotransmitters.
Sparse networks are not understood to anything like the same extent as dense matrices. And another key property that most ML is missing is large numbers of feedback loops. Again, that makes predicting behaviours extraordinarily difficult.
It couldn't do much mainly because we don't know what it should be doing (to emulate brain).
In this case, the software challenge is far greater than the hardware challenge.
> We estimate that a system with 10^11 active 10μm x 10μm elements (comparable to the number of neurons in the brain) all firing with an average pulse rate of 1KHz (corresponding to a high neuronal firing rate) would consume about 50 watts. The quiescent power drain for this system would be 0.1 milliwatts.
Note they are referring to 10μm process technology. Modern state of the art technology would probably get the power consumption of such brain scale system down to under a single watt.
However, the effective network can route dynamically (by faking things on top of an initially-zero-weight all-to-all connection pattern between two neuron populations). One of our PhD students is working on this, and on the types of dynamic online learning that this enables, modelling the dynamic generation and removal of synapses that occurs in biological neurons. We also support tuning of connection weights in response to the history of synaptic activity via Spike Timing Dependent Plasticity (STDP), and have done for a few years (using earlier generations of the hardware config).