A 'Brief' History of Neural Nets and Deep Learning | Hacker News Reader

A 'Brief' History of Neural Nets and Deep Learning

andreykurenkov.com

245 pointsandreyk20 comments

Excellent article, thanks for the contribution.

Perhaps some independent validation, but I was coincidentally having this conversation the other day with a relatively well known computer vision researcher, about why it seems like the idea of neural nets has floundered for decades and suddenly it's the hot topic, and we're seeing massively improved results.

His answers, summarized, are that:

1- Big data is making possible the kind of training we could never do before.

2- Having big data & big compute has made some training breakthroughs that allowed the depth to increase dramatically. The number of layers was implicitly limited until recently because anything deep couldn't be practically trained.

3- The activation function has very commonly in the past been an S-curve, and some of the newer better results are using a linear function that is clamped on the low end at zero, but not clamped on top.

All really interesting to me. This is making me want to implement and play with neural nets!

Of course, now the big question: if we have a neural net big enough, and it works, can we simulate a human brain? (Apparently, according to my AI researcher friend, we're not there yet with the foundational building blocks. He mentioned researchers have tried simulating a life-form known to have a small number of neurons, like a thousand, and they can't get it to work yet.)

I agree with everything but the last. Simulating a human brain is perhaps necessary for consciousness uploading, but not for super human ai. We have been able to travel faster by machine than foot for a long time, in spite of not being able to build a robot that can simulate human running.

> Simulating a human brain is perhaps necessary for consciousness uploading, but not for super human ai.

I'm not suggesting otherwise, and what you said might be true, I don't know. But honest question: how can we know it's true before we can do either? How can we suggest neural nets are a good building block of super human AI, if we can't verify we can do regular human thinking, or sub-human thinking?

Because the neural net is modeled after (simplified) brain neurons in the first place, it seems reasonable to suggest that if we can't simulate a simple brain, then we can't validate that the model is correct, right? It might be very productive, and it might 'work' in some sense, but we don't know whether it can truly act as the lego brick of brain building material until we can build a functional brain.

Your observation is true, and worth considering. Machines can travel faster than humans, but that's different in part because we didn't start trying to simulate the human foot, right? The computational equivalent would be that machines can multiply a lot of numbers much faster than a human. That is super-human calculation, and it is mechanical thinking, of a sort, but most people wouldn't say it counts as "AI", perhaps the same way that most people wouldn't say a car counts as human running, even if it is faster.

Neural nets are being used to classify images, find objects, identify people's faces in a crowd or in difficult to see situations. But, they currently can't tell you to stop classifying images because you're asking the wrong question, or interrupt you to say you're looking great today.

Before we get wildly over-optimistic, as every HN thread on AI becomes, realize that neural networks have basically nothing to do with actual biological neurons. There are a bajillion things neurons do that neural networks do not do. And there are similarly many things that are a part of neural networks, that biological neurons do not do. In addition to the fact that neuroscientists still do not have a clear understanding of biological neurons: you cannot expect to reverse-engineer something you barely understand yourself.

This is not entirely true, as there are spiking neuron nets and neuromorphic chips (like IBM's TrueNorth) that emulate what we have so far understood about how neurons operate. But it's true that even these are simplified models and as far as I know cognitive science has not explained the higher level operation of the brain. In general I think breaking down how modern deep learning works makes it seem unlikely that it will lead to true AI without additional insights, so I am unsure why very intelligent people such as Elon Musk make it seem like an imminent threat - not to mention the idea is predicated on continued exponential increases in computing power. But the stuff going on in AI right now concerning image comprehension and question answering is fairly impressive, so I think it is feasible we will have impressive AI assistants and the like within a decade.

"This is not entirely true, as there are spiking neuron nets and neuromorphic chips (like IBM's TrueNorth) that emulate what we have so far understood about how neurons operate."

How is this related to the neural networks we are discussing? (deep ANNs?) As far as I know, the ANN topic in machine learning, apart from its origins, is completely unrelated to the simulation of biological neural network models: https://en.wikipedia.org/wiki/Nervous_system_network_models

They are often confused because of the name and because ANNs did begin as extremely simplified models of biological neural networks, but the machine learning concept that is setting all the records in vision / speech recognition serves no purpose as a model of a biological neural network.

the machine learning concept that is setting all the records in vision / speech recognition serves no purpose as a model of a biological neural network.

I think you are overstating this. There are definite links between the two, even if ANNs end up different because of the tools we have available to us.

CNNs were explicitly designed to mimic the behavior of the visual cortex.

Most of Geoff Hinton's career has been built around thinking very hard about biological computation.

One of the major criticisms of back-propegation in neural networks is that it is biologicaly implausible.

https://en.m.wikipedia.org/wiki/Convolutional_neural_network...

> Before we get wildly over-optimistic, as every HN thread on AI becomes, realize that neural networks have basically nothing to do with actual biological neurons.

Yes. I think I was already fully agreeing with you, and suggesting the same but in softer language. Maybe you meant that for the parent or more generally the thread, but I think that I should add something: I may have unintentionally conflated the discussion on neural nets and the biological research my friend was talking about.

My limited understanding of what he said - and I'll go ask him for a reference - is that we (as in scientists somewhere, not me personally) have come to a more or less clear and complete understanding of the chemical and electrical processes in the neural functioning of these simple life-forms, from top to bottom. (I don't know that's true, but that's what I think I heard.) They then tried to put together a complete simulation of this simple lifeform's neurons. This simulation, as I understand it, is not a neural network per se, but something trying to be much closer to a biological simulation. And when they turn it on, apparently, it doesn't work.

That story, if true, tends to confirm what you said; we're missing something in our understanding of biological neurons. Which, I think, lets me say more confidently that we can't suggest we have the building blocks for AI yet.

We are making very similar points I think. My main point is that it is not a given that we must simulate a human brain to get super human AI. We may, we may not, we dont know, neural networks or otherwise.

I'm of the opinion that you're absolutely right!

And as such, I don't know, but I do currently believe that we don't have the computational foundation for AI yet. My friend, who's done more AI research than I have, and who's probably smarter than me, said he thinks we probably have everything we need for the logic part, and the only thing missing is enough computation and enough data to simulate the amount of input a human gets. I was surprised by this response and pressed him on it. "It's turtles all the way down."

>Of course, now the big question: if we have a neural net big enough, and it works, can we simulate a human brain?

There are many possible neural networks. Most of them are not human brains.

He mentioned researchers have tried simulating a life-form known to have a small number of neurons, like a thousand, and they can't get it to work yet"

302 to be precise - https://en.wikipedia.org/wiki/OpenWorm

Upvoted -- what a nice and detailed history.

Thanks for linking to the old NYT article on Frank Rosenblatt's work. One can see how researchers of the time were irked by delirious press releases when the credit-assignment problem for multilayer nets had not been addressed.

(We managed to mostly address the credit-assignment problem for multilayer nets...but the delirious press release problem remains unsolved.)

Incidentally, it's "Seymour Papert", not "Paper" (appears twice).

Thank you, I will correct the typo.

This is too long ago for me to know what actually happened. Both Rosenblatt as well as Minsky and Papert knew that networks could perfectly learn the XOR function. See https://en.wikipedia.org/wiki/Perceptrons_(book) and also stated in this article.

I knew there was a fight between Minsky and Grossberg at that time. Perhaps there have been other reasons that are not so well known that led to an AI winter. Have these winters ever be quantified though?

For the article, you can find it in here: http://msrvideo.vo.msecnd.net/rmcvideos/258318/dl/258318.pdf

Great read. Its incredible what we, as a species have been able to achieve in the last 2 centuries. I feel like we are where scientists were with computers in the 1950s. We are starting to see the big picture, but its wide application is still decades away.

Great article, took an neural network course while I was doing my graduate study long time ago and it might be time to resume that subject. the NN training then took a long time to be impractical for real use and now it should be much faster.

Author's done an excellent job of explaining what the problems were at every stage and how NNs evolved to solve them. Learnt a lot from this series.

This is actually a very important chapter in human development. And it will pass in the blink of an eye. Works like these are good, important.

just to re-iterate, the reference text #13 I.e. parallel distributed processing (vol. 1&2) are also an excellent introduction to the field, from its infancy.

contains a collection of papers by nn luminaries including rumelhart, Hinton etc. Very highly recommended.

Reply on news.ycombinator.com