Computational Power Found in the Arms of Neurons
quantamagazine.org
quantamagazine.org
I mean there are unicellular creatures that can do incredibly complex things, and neurons, as all cells, are ultimately the descendants of such unicellular organisms, "teamed up" into a multi-cellular creature. They have a lot of internal structure, organelles, whatnot. Modeling them as some simple switches seems a stretch.
My point is, overall this doesn't seem so surprising to me as a layman. (Which doesn't mean more than just literally that. I'm not claiming laypeople should decide scientific questions from their armchairs.)
Plus, the sum and threshold model actually gets you pretty far, as evidenced by artificial neural networks, so there might be some resistance to adding complexity that might not be necessary.
Edit: Google "dendritic computation" if you want to dig deeper.
Edit: i am blocked from posting, so here are the refs:
https://www.ncbi.nlm.nih.gov/pubmed/20800473
https://www.ncbi.nlm.nih.gov/pubmed/25554708
https://sci-hub.tw/https://www.sciencedirect.com/science/art...
This[0] research shows a single neuron can compute XOR, thus the artificial neuron has less computational power than a real one.
[0] - https://www.reddit.com/r/MachineLearning/comments/ejbwvb/r_s...
The potential for computation inside each neuron is similar to recurrent neural network.
And to be honest, it took us somewhere. Yes we don't have AGI or anything close to it but the products of machine learning is something we depend on every single day now.
It still works, but it might not be a useful model to anyone studying the brain. I've yet to meet the neuroscientist who assumes it is, so perhaps that's not a problem.
It's a possibility, but I sometime have the feeling that people dismiss the idea that such a simple model of the brain could be enough to explain complex behaviors because they want to believe there is more to the brain.
I'm sure real neurons are very complex and difficult to model, but I also believe that the real challenge is to explain how neurons interact with each other, not how they behave individually.
As it is, ML is running away rather fast from the integrator model by introducing explicit gating and nonlinearities in the neurons.
Yet it is still a toy compared to a nematode.
But models based on incredibly simple neurons can already produce quite complex behaviors. They show how many simple computing units interacting with each others can lead to things like vision. And I do believe that this is a fundamental principle.
Maybe we should explore that idea and scale this model up, instead of rejecting it as "too simple" and hoping that the complexity of the brain will be fully explained by the discovery of some quantum effect in neurons.
> As it is, ML is running away rather fast from the integrator model by introducing explicit gating and nonlinearities in the neurons.
I think the idea of a non-linear activation function has always been around. But for the rest I agree.
https://www.quora.com/If-a-neuron-were-a-galaxy-how-would-th...
The arms of neurons have a lot of the surface area, don’t they?
ANNs are abstractions that seem to work. We still don't have a full understanding of how two neurons communicate.
The other things we figured out in the meantime relate to different types of neurotransmission (e.g. volume neurotransmission), active processes in axons and dendrites like vesicular trafficking, synaptic tag-and-capture (if it turns out to be true) and all kinds of weird types of plasticity. Basically a neuron's function is a lot more nuanced than the simplistic "integrate and fire" idea.
Artificial neural networks (in the deep learning sense) therefore don't really have much of anything to do with biological ones function-wise. It really is just regression with a lot of nodes and layers, and maybe some bells and whistles. That doesn't mean they're not powerful in their own right, just not comparable to brain circuits and don't do the same kind of thing nor solve the same kinds of tasks.
https://senselab.med.yale.edu/ModelDB/showmodel.cshtml?model...
The simulator used is called Neuron. Code for it is written in a custom language called Hoc and models are implemented in yet another domain specific language. Hardcoded parameters galore including whole sections of code that are prefaced with: this has been copied from this other publication.
https://senselab.med.yale.edu/ModelDB/showmodel.cshtml?model...
The dendritic tree geometry is specified as hardcoded set of point and branch declarations
https://senselab.med.yale.edu/ModelDB/showmodel.cshtml?model...
(apparently that morphology was reconstructed from imaging data?)
Without wanting to spend a lot of time on it, why is it horrific?
There's nothing intrinsically wrong with hardcoded values as they're biological constants, and nicely identified by the standard convention of using ALL_CAPS_VARIABLES.
Edit: Scanning a bit more, he's probably referring to this file, which is pretty horrific:
https://github.com/ModelDBRepository/254217/blob/master/_mor...
Even if there are issues, it's tremendously valuable that they shared this, so others can reproduce and build on their work. That said, most scientists would like to have better quality code in science as well, but I think the incentive structures and other demands that make it challenging...
I think a mass open source campaign to replace NEURON and ModelDB with something more modern, and clean up and convert every published model would be worth the effort. There are quite a few alternative programs (GENESIS, Neuroconstruct etc) that are either incomplete or abandoned because of lack of funding/interest.
Neural networks are coarse grained models of large scale brain structure. The fact that these models can be taught to do very interesting brain-like things (especially pattern recognition) demonstrates that this structure is important and fundamental, but that doesn't mean it's the whole picture of what's going on in the brain.
I say talking into the wind because in my experience most CS people tend to hand-wave away biology at the cellular level and below. There's this zealotry around us being "almost there" with AGI and the rest that blinds people to the real magnitude of the problem and how much is really happening in the brain. This likely includes a whole lot that we haven't even started to really understand.
It has been suggested that neural networks have diverged from biological principles to the point where biological research does not provide any useful improvement to current machine learning techniques. Back propagation of errors was a major advancement in the 1970s, and it was developed purely based on mathematical principles like statistics, differential equations, and calculus. As far as I have read, error back propagation is not how biological NNs learn, and it seems to be a much more efficient strategy. Biology seems much more brute force in comparison.
Biology is more decentralized for sure, but I wouldn't be so fast with calling it "brute force". One interesting feature of biological neural networks is the variety of neurotransmitters. In very simple terms, they are not only capable of exciting or inhibiting neurons from firing, but they are also capable of regulating how neurons adapt to the stimulae they have been exposed to. In other words, the biological network not only learns, but it runs the learning algorithm itself. It appears to do this by creating various layers of communication with different effects on the computing nodes.
In my view, we have not been smart enough yet to figure out nature's algorithm.
I find this is a conveniently simplistic view of the current state of machine learning, which they wiggle out of by saying "traditionally built". Anyone working on ML and neural networks these days knows that modern networks are built out of building blocks -- certain standard architecture archetypes (MLP vs CNN, autoencoders, GANs, etc), resnet blocks, training regimes (layer pre-training, progressive training, etc), normalization methods, and so on. Sure, the term "neural networks" originally implied that each linear-non-linear operation pair represented a single neuron, but it's not a stretch to say that if neurons are more powerful than we thought, then each neuron represents, say, a small MLP block, or a resnet block. The biological analogy still stands. Citing ideas of ANNs from two decades ago to promise how new results in biology may change "current" thinking in ML (by citing decades-old ideas while calling them "traditional" no less!) is disingenuous.
A singly honeybee with its billion synapses can adapt to a variety of situations and environments, learn on-the-fly, perform complex sequences of tasks, cooperate and communicate with other bees. All of these capabilities are emergent and packed in its tiny head.
State of the art artificial neural networks (ranging beyond billions of parameters now) only do the thing they're specifically built for, only after training with bazillion specific examples and consume tons of energy while doing so.
Which one of these sounds like the brute force approach?
It doesn't matter whether their behavior is considered "instinctual". What matters is that they can quickly adapt their behavior to entirely novel scenarios:
```The dendrites generated local spikes, had their own nonlinear input-output curves and had their own activation thresholds, distinct from those of the neuron as a whole```
How would this work in practice? Apply activations to multiplication values of the weight, or just don't perform the multiplication if the activation of the node is low?
Which is what's being done lately, but then we know next to nothing about the topology of real neural nets nor electrochemical communication inside a neuron.
> This paper is amazing. What is missing from the description above is that this is the first example of how human neurons are qualitatively different than rodent neurons (not only more computation power, but categorically different computation).
> ELI5: the way the biological human neuron implements XOR is by a formerly unknown type of local response to inputs, which is low below the threshold, maximal at the threshold and decreases as the input intensifies above the threshold. We never saw anything like that in any other animal. (link to the relevant figure from the paper)
Rodents being presumably the go-to non-human object of study.
Here is the reddit comment: https://www.reddit.com/r/MachineLearning/comments/ejbwvb/r_s...
In Creatures Evolution Engine games such as Creatures 3, each neuron and dendrite is a fully functional register machine.
https://www.biorxiv.org/content/10.1101/613141v1.full
They model a single cortical neuron as a deep neural network with 7 hidden layers consisting of 128 hidden units each.
Biological neurons are A LOT more powerful than "neurons" in neural networks. If you hear a claim about computer-brain parity being close - the people making it almost certainly don't understand the power of cortical neurons.
I guess I'd say a Turing machine is my definition of a machine.
And I suppose the implication is that if we are machines, operating on a stream of input, does that make us deterministic?
EDIT: What I'm ultimately interested in is if we, as humans, are operating in a realm that logic cannot operate in. If that makes any sense? I'll probably have to define realm!
>What I'm ultimately interested in is if we, as humans, are operating in a realm that logic cannot operate in. If that makes any sense?
Any mix of logic and randomness is still a machine. You must assume something spiritual if you think there exist rational action without computation behind it. Randomness does not help there.
I'm still uncertain i.e. spiritual. I'm finding, as I grow older, logic and reason fail to answer for my experiences. Whether that's a shortcoming in my own ability to comprehend logic and reason, or because logic and reason simply do not have all the answers, I do not know.
> Randomness does not help there.
The human condition, one might argue, emerged as a way to handle randomness. I suppose, at some level, if we were to remove all randomness from the universe, life would be pretty pointless (insomuch as life is not already pointless.)
There is no purpose of life, in my opinion. But with randomness it's a lot more intresting.
Ultimately you'd find a common unicity — like electrical charges, strong/weak force, arrangements of "gates" and "structure" etc. — but used in different ways (for instance iirc charged clouds of gas in space don't form new stars because gravity at close range is weaker than the charge that repels them, only strong enough at larger distances to hold them together; that's dramatically different from the use of charges in biological cells).
The cosmos itself is but a big machine. Who's to say we (I mean the whole planet, maybe star system, maybe galaxy itself) aren't actually just a single "cell" of the cosmos? That we are part of a much bigger machine, that we are like those processing units on dendrites in the article, if the universe is a big brain of sorts?[1]
These are all types of machines (X. process of information; Y. engines to convert energy; Z. structures that "restrict" "flow" like gates, transistors, cell membranes, dendrites; N...), which apparently may be expressed, materially implemented in different ways.
To take your question about logic.
If you mean the universal logic exposed above, I don't think so personally — merely because there's no evidence for it whatsoever, no observation; and there's also no need if we accept that from such "machinery" complexity may emerge complex systems (e.g. humans).
If you mean logic from within the human mind — and this begs the question of whether maths and physics are "invented" or "discovered" in the background — then we must assume a subjective answer, at best an aggregate of "non-disproved facts" that we can all agree on within normalcy (edge cases helping us understand said average norm).
"Logic" is but one of several operating modes. Are we more than that? Most certainly yes. That's the experience of all of us. Because perception is so centered on thought, we tend to overappreciate the importance of thinking in our lives, in our behaviors and even values; but the relative picture gradually revealed by biology and psychology and sociology and economics etc. is that we are mostly irrational, mostly automatic (trained habits, patterns recognition, educated intuition, etc), and actually very little in the way of "logical behaviors units".
I'll let you ponder the discrepancy from an absolutely (perhaps) low-level logical machine to the possibility of a chaotic non-logical emergent high-level behavior. What it means for life, for AI, and possibly much bigger or smaller things we consider "inert", for now.
[1]: I mean, look at the larger structures of the universe, tell me it doesn't look like a sample from a biological tissue... http://cosmology.com/GalacticWalls.html (figure 6: http://cosmology.com/images/darkmatterdistribution.jpg)
Very true. Thought can become our master easily. As a programmer, I often find myself struggling to get out of 'logic' mode or 'thinking' mode.
> I'll let you ponder the discrepancy from an absolutely (perhaps) low-level logical machine to the possibility of a chaotic non-logical emergent high-level behavior. What it means for life, for AI, and possibly much bigger or smaller things we consider "inert", for now.
Wonderfully put! I think the ideas of emergent behaviours are a saving grace for the incessant desire of ours to reduce everything to its atomic (in the sense of an atomic operation, not an atom) components. The emergent behaviours, can't be predicted. They can't be algorithmized. Becuase to build algorithms one must go down to those atomic operations.
My wider concern is that we are currently obsessed with analytical thought and philosophy and that the continental philosophy has become a 2nd class citizen. I think this has happened becuase analytical thought lends itself to being algorithmized, whereas continental philosophy does not.
Sorry, that was a bit of a tangent!
Yes! Love the pictures! I have had the same thought/feeling when I first saw them too!
Very well said! Humans think a lot in terms of dichotomies (here the micro/macro, or whichever scale you want to consider), but I've heard biologists and physicists explaining that the closest to fundamental functions is closer to e (natural log), sinus (and hyperbolic stuff), the notion of "binary polarity" is very specific and quite limited — look at the Standard Model, it's clearly more complicated.
I personally think that's how the dimension of time emerges in our perception (any perfect observer): periodicity in all phenomena (but with e.g. Fourier it gets extremely complex at emergent thresholds), the arrow of time (non-periodicity is basically high entropy, 'heat death'/homogeneity).
Cue any such human-driven dichotomy (I think this one is relatively easy: the deepest processing system is probably close to "emotions", and these at the lowest level work in terms of "rather good" and "rather bad" (different sub-regions of the brain) + some "general integration" (third region). And what do you know, we tend to be heavily polarized in general, it's like most people feel reassured (familiarity) when things are explained in terms of black and white, left and right, ones and zeros.
(Just my thoughts on it.)
> My wider concern is that we are currently obsessed with analytical thought and philosophy and that the continental philosophy has become a 2nd class citizen. I think this has happened becuase analytical thought lends itself to being algorithmized, whereas continental philosophy does not.
Strongly agreed. I think we're witnessing a kind of "revival" through many fields (some spiritual, some historical). Things are moving. I think dumb things like numbers also influence people, it's been ~20 years since the turn of the millenium and that's enough for 1 generation to weigh in and the others to accept change because "oh new era, obviously, new number!" — this plays at the subconscious level of very crude processing, me thinks.
Nice tangent, hopefully not too hyperbolic. ;-)
Although I don't see how it would make a difference to this question where in the brain the computational power arises from.