Comment by pessimizer | Hacker News Reader

Comment by pessimizer | Hacker News Reader

Parent Full thread

This is a question for anyone who knows more (or reads more) about neural networks than me. My problem with the perceptron since I was a kid is that it doesn't seem to model how neurons work very well at all. It creates a neuron that exists in a timeless place; kind of a homo economicus for cognition.

My understanding of neurons: The signals arrive at the dendrites at different times, as soon as a signal arrives its strength decays steadily, and when the sum of decaying signals and an additional signal reach a threshold, the axon fires, getting rid of all energy and beginning the process again (creating the largely irregular pops that everybody is used to listening to.) Aside from that, there are effects from other bodies and chemical concentrations between neurons that are mostly unknown (that I get leaving out as not central to the process.)

My question: is there any paper that explains how that process, which seems pretty easy to model, can be mathematically reduced to perceptrons (hopefully comprehensible to the educated layman) which are just addition and a condition?

Look into computational neuroscience, neural engineering, spiking neurons and spike-timing dependent plasticity (STDP). These are more accurate models of biological neurons and their synaptic interactions.

From the Hodgkin-Huxley (HH) neuron model to integrate and fire (I&F) you can approximate and simulate various levels of realism in artificial neural networks. There's even the Nengo library[0] which can simulate these for you in Python. Unfortunately, they're very hard to work with compared to perceptrons, artificial neural networks, and deep learning (see Neural Engineering[1] for a good framework). Additionally, they're far more computationally intensive with the HH model coming in at over 800 FLOPS to the I&F model (which lacks a lot of the behavior and dynamics of biological neurons) which has 7 FLOPS. Then try representing numbers with spiking neurons... You need several dozen, working as a population[2], to accurately represent a single floating point because they're so noisy. This makes working with data pretty difficult and expensive.

[0] https://github.com/nengo/nengo

[1] http://compneuro.uwaterloo.ca/research/nef.html

[2] https://en.wikipedia.org/wiki/Neural_coding

First off preceptrons are very simplified models, a first pass really at modeling brains in silico decades ago.

There is a whole field of theoretical neuroscience dedicated to this problem, and have made quite a bit of progress, still a bunch of work left to be done.

The classical introductory text on the topic is the book by Peter Dayan and Larry Abbott called computational and mathematical modeling of neural systems.

[deleted]

Thanks for the reference.

edit: Thanks very much; if that doesn't contain what I'm looking for, it'll give me further places to look.

There are much better models out there for modeling networks closer to the biology. The current generation is kind of understood to be an extremely rough model that draws rough inspiration from biology. But the important thing is that Ann models are performing better on image recognition and nlp tasks than any other techniques. The fact that they demonstrably work is far more important than biological mimicry. There is another model called a spiking neutral network that basically aims to mimic biology and incorporates time and usually many more neurons and sparse connectivity. While this is interesting, so far they haven't produced anything too mind blowing and due to the increased complexity they are much less computationally efficient right now. But those actually interested in the biologically inspired aspect are streare stream to

Thank you very much, this gives me a lot more to read.

Artificial neurons are a model rather than a simulation.

McCullough & Pitts sought to computationally model neuron's information processing ability, not simulate their biology. Their 1943 paper "A logical calculus of the ideas immanent in nervous activity" invented the Artificial neuron.

"Because of the "all-or-none" character of nervous activity, neural events and the relations among them can be treated by means of propositional logic." - Mcullough & Pitts 1943

Rosenblatt's Perceptron's tunable inputs & Werbos's Backpropagation allowed the Artificial Neural Nets Hebbian learning.

Minsky & Papert showed a Multi-layer perceptron can learn non-linear functions.

It is quite interesting that you bring up Hebbian learning, which at its heart is attempting to endow a model with a form of memory stored in the connection or 'edge'.

Current state of the art systems (RNNs, LSTM and ResNets) are taking this one step further attempting to create dynamic memory (much like we believe neural networks work biologically for short term memory storage).

I wonder if the next steps for models are to formulate mechanisms for actually modifying their own graphical structure with a sort of 'protein' based memory, much like the brain is believed to store long term memory? Does anyone have/ or has seen any papers on the topic? Could you provide refs?

Yours is a good characterisation of Hebbian learning.

Backpropagation in ANN & Hebbian learning in NN analagously strengthen correlations between inputs & succesful outputs - learning useful associations.

Stanley & Riskonen's NEAT evolves neural structure http://nn.cs.utexas.edu/?neat

Schmidhuber & Togelius have nets that prune connections to evolve / learn structure.

Thank you for the references.

I was not as much interested in mechanism to reduce the training speed by optimizing the structure in order to learn more efficiently. NEAT aims to 'minimize topologies and grow incrementally.'

Rather my interest was in something similar to Q-learning mechanism or RL-LSTM that could learn structural sub-domains within a network, and reimplement them in other portions during training (maybe paralleling NEAT) or during instances of poor output probability.

There is some evidence that large deep nets evolve using convergent strategies and solutions across modalities - at least for words and pictures.

Low level specific edge & feature detectors combine into parts of things, these then combine into thing detectors and ultimately vectors representing objects and encoding their relationships. The final layer classification ( the training objective ) is a low dimensional slice of a rich conceptually navigable semantic vector space.

That these top layer 'thought vectors' occur even for limited classification objectives implies the structure comes from the datas contextual relationships and the net learns how things relate to better classify what things are.

Here's an online book I read that explains perceptrons and sigmoid neurons from the ground up. I think it's a great resource and might help answer some of your questions.

http://neuralnetworksanddeeplearning.com/chap1.html

This chapter also takes you through creating a handwriting recognizer in Python without having a package deal with the internals for you.

Correct. The similarity between actual neurons and "neural networks" exists in the name only, functionally they are night and day different. Neural nets are really closer to multi-variate regression curves than brain matter.

There are arguments that you'd need to create a computer capable of doing "infinite" precision real computation to recreate this effect.

Where do those arguments come from? I always thought that the problem was that it might be an irreducibly parallel process, and would need its own hardware to simulate cheaply.

That does sound interesting; any links or cites?

Reply on news.ycombinator.com