Comment by rm999 | Hacker News Reader

Comment by rm999 | Hacker News Reader

Parent Full thread

>Deep learning, modeled on the human brain, is infinitely more complex [than machine learning]. Unlike machine learning, deep learning can teach machines to ignore all but the important characteristics of a sound or image-a hierarchical view of the world that accounts for infinite variety. -Yann LeCun

I strongly disagree with a lot about this quote, even though it comes from a brilliant man I highly respect (my thesis research was inspired from some of his older work, and in my job we work with techniques he developed around deep learning). What I dislike is it utilizes hype to make deep learning seem mystical; in actuality it's a natural extension of old techniques that clearly fall under "machine learning".

Deep learning is a neural network with 3 or more layers instead of the 2 layer networks that were developed in the 1980s. People tried 3 layers back then, and they didn't work well. Yann LeCunn and other researchers found cool ways to get 3+ layers to work in the 90s and again in the mid 2000s. More recently, researchers have just thrown a ton of data and computational power at them to get them to work. But fundamentally this was an extension of established techniques. This article that recently hit the front page actually breaks it down really well: https://blogs.nvidia.com/blog/2016/07/29/whats-difference-ar...

I think my main point here is that deep learning is quite accessible to people who are learning machine learning. It's great at solving some really complex problems (that can certainly resemble true intelligence), but is not the right tool for other problems.

Viewed strictly as a neural network, how many layers deep is the human brain?

Neural networks are based on a very, very outdated (and incorrect) model of how a neuron works, so your question is roughly akin to asking "how many vacuum tubes are in my GPU?"

Mammalian brains are so densely packed that we don't really know what their internal architecture actually looks like--that's why estimates of the number of neurons keeps increasing over time [1]. Most of our models are based on some autopsies, and rely on things like fMRI or electroencephalography (not often you get to spell that word!) to work out which portions of the brain are involved in certain tasks. We have no hope of being able to build a connectivity graph between neurons at our current state of technology.

The short answer is that the cerebral cortex is usually identified as having 6 layers. These can't be correlated with layers in a neural network, however. I do seem to recall a spitball number of 30-ish in relation to the visual cortex, but I've been unable to find any references as to what that number would actually represent, let alone if it's been superseded.

[1] The current estimate is around 100 billion. If you want to know the number involved in, say, the visual cortex, that appears to be on the order of a few billion.

> The short answer is that the cerebral cortex is usually identified as having 6 layers. These can't be correlated with layers in a neural network, however. I do seem to recall a spitball number of 30-ish in relation to the visual cortex, but I've been unable to find any references as to what that number would actually represent, let alone if it's been superseded.

That short answer is very misleading. Those 6 layers are histological layers of density and structure in the cerebral cortex (eg how the cross sections look a little different). You might have many (30+) layers of cells with high interconnectivity in each of those 6 layers. Current estimate is 100 billion for the number of neurons, but the estimate of inter-connectivity is on the order of 100 trillion synapses. The human brain complexity many orders of magnitude higher than anything we have ever "simulated" in silicon.

It's not misleading because they were careful to point out that these 6 'layers' have nothing in common with how layers are defined in Neural networks. In Neural nets, layers are effectively subgraphs that are bipartite. They can also be computational stages in some networks like convolutional.

The brain's layers are both much more loose in their allowed interconnections but also more modular in their functionality. There are different visual areas (usually 5) at different locations of the neocortex that connect to each other (feedforward and back) in complex ways. As residents of the neocortex, each visual area will thus have 6 layers which are in turn dividable into sublayers (often 1 - 4). These layers are structural but are also partially organizational tools for the scientists.

Thanks, this is good detail. Would you recommend any reading on the up-to-date understanding of brain architecture?

Easy, each transistor represents a tube. QED

Brain doesn't work like NNs in software. It's different beast you can't really compare layers in one and in second.

There are several reason why this model just does not apply to the brain[1]:

1. The brain is not organized in layers this way. Instead there are interconnected circuits, feedback, brain regions, bidirectional connections, and such.

2.The brain's connections are feedforward only without "error feedback".

3. The brain doesn't learn from training data with correct answers (or a "ground truth").

4.The neurons in the brain spike and use different learning algorithms.

[1]https://www.quora.com/How-many-hidden-layers-are-in-the-neur...

Just wanted to add that there _are_ certain portions of the brain which do seem to be layered, namely fundamental image-recognizers in the visual cortex. These were a lot of the inspiration for layered neural nets, and may explain some of their success on similar tasks.

As I point out in another post (currently above) the layers of Neural nets have nothing in common with those in the brain. Only confusion will be met in heading down that path.

What is had in common, and what was inspirational were (harsh) experiments on kittens, done in the 60s which provided hints as to what concepts were important in the primary visual cortex. These experiments lead to ideas of pooling and averaging as ways to calculate invariance, which made their way into the neurocognitron and then the convolutional net. The concepts of thresholding units and summation of inputs also played a part in the designs of Artificial Nets more generally. Most recently, new artificial NN designs have been loosely inspired by the importance of attention in biological networks.

Another similarity, but this is a general computational similarity, is the importance of hierarchical organization in easing learning difficulty. But unlike in the mostly feedforward networks we're used to in ANNs, brain's networks are also feedback, or with backward projections, across areas and up and down hiearchies and so on.

But there is more to vision than V1 and V4. The visual cortex has more arbitrary connections across layers and is not restricted to the bipartite structure common in MLPs and is more fine grained than the hidden layer summation across timesteps of RNNs.

Other things the visual cortex computes but not well captured in models: speed and directional selectivity, nontrivial color transformations, temporal frequency, global v local + foreground v background, optical flow, motion tracking, inhibitory neurons, higher level scene understanding, feedback connections, perceptual color constancy and contributions to memory formation.

Yes, I was specifically referring to the findings from those kitten experiments.

Of course there are plenty of other things that the visual cortex is involved in, as far as we can understand it at least. I'm talking purely about the macrostructures that are involved in elementary vision functions like orientation.

I'm in the camp as well that conflating brains and ANNs is a dangerous and hype-prone line to walk, but I'm not sure you can categorically say that the layers in V1 and the layers in a CNN have nothing in common, with respect to the computational capabilities they model. Likewise with convolutions and columns.

I say that the computations in V1 and the computations in the CNN share a fair bit in common.

However, I doubt that the layers in V1 and CNN have much in common for a couple reasons. First is because the layers in V1 will differ depending on who you're reading from and when they wrote it. The layer divisions are more structural than functional, so far as we know. You can't trace ever more abstract features as you move across V1 layers, not least because V2-V4 are also involved in this.

Next is that V1 sends projections to "higher" and also "lower" areas while connecting to other areas than just the visual. Within V1, connectivity from lower to higher layers are not obvious. Some inputs from the LGN might feed into layer 4 and others to differing layers depending on whether the computation is motion or color etc specific. Layer 5 might also get connections from more distant areas. This level of interconnectivity suggests that very different computations are going on even at the level of just V1. The connections are finer and more intricate than simple bipartite divisions. Basically, they're working out more than just features that are robust to rotations and translations (e.g. speed, direction, dual stream signal integration, oscillations...).

The prevalence and importance of feedback and recurrent connections is not yet well understood but are believed to be important in the sort of high level scene understanding that are still open problems.

And related to that, the notion of inhibition is turning out to be more and more important in biological networks though they aren't really key to any neural nets. Some of their repeating patterns or motifs, have a place in directional and size selectivity but more intriguingly, are figured to be key in the top down flow of information that allows for our (mammalian) complex scene understanding.

The importance and whether there is any function of the column is not without controversy, it may well be an artifact of what tools were used in the early days. The brain is very flexible in its organization and attempts at organizational structures such as columns aren't very consistent even across individuals of the same species: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1569491/

I also believe that some[1] believe that CNNs and other image processing networks do some extra work that our brains don't need, as we can "cheat".

For example, we get some translation-invariance for free with the ability to move our eyes.

[1] http://ilp.mit.edu/event.jsp?id=6094 (I had the video URL bookmarked, but it's broken now)

That is a good explanation. I'll point out for anyone that clicks through, that unlike what he says, depth in ANNs do serve a purpose: to make learning more tractable by building representational abstractions.

There are feedback and recurrent connections in the brain but what (2) means to say that the specific mechanism is not biologically plausible. I know there are prominent Deep learning researchers who are trying to work around this but those explanations have never struck me as particularly parsimonious. It's better to start an explanation from experimental data on brains than from them most important aspect of neural nets.

The human brain is what they call a recurrent neural network which means that it contains directed cycles. Classic neural networks are strictly hierarchical and one-way. If you take into account that signals can keep resonating in the brain (firing in circles) then the amount of layers is infinite.

Recurrent neural networks are incredibly hard to train with the exception of liquid state machines (also known as echo state networks or reservoir computing).

I remember reading that the fastest human reflexes and computational outputs seem to happen around 100ms, and it takes roughly 10ms to transmit a signal from neuron to neuron, so that would imply 10 layers deep is enough for, for example, human level visual recognition. Of course, the human brain is highly recurrent, so it gets much more complicated for any higher level thinking. And it's also much, much wider than our neural networks, even if not spectacularly deep.

https://en.wikipedia.org/wiki/Cortical_column

https://en.wikipedia.org/wiki/Neocortex#Anatomy

In terms of number of neurons, human brain is around 10^11. Current image classification deep networks are around 10^7. However, its not just the number of neurons, the connections per neurons also matter. The connections per neuron in a human are in thousands while in modern networks they are in hundreds.

The human brain is also very efficient so it also makes a very good use of those neurons, and modern networks aren't. However, a human brain trains over a very very long time and also prunes itself constantly, while such efficiency techniques at scale for very large networks are being explored in deep learning only recently.

I find that to be a very interesting question.

It contains feedback loops for a start, so the notion of depth may be misleading.

> Deep learning, ... is infinitely more complex

When I read this I interpret it to mean they have no idea how deep learning works. (More specifically: why it works as well as it does.)

And the part about being modeled on the human brain just seems to be a fortunate (?) coincidence.

If you mean the lack of theoretical guarantees due to a highly non-convex optimization, you could argue the same thing about 30 year old 2-layer networks. And we actually do have a decent idea why deep learning works well:

http://arxiv.org/pdf/1412.6544v4.pdf

http://arxiv.org/pdf/1412.0233.pdf

Thankyou that looks awesome.

Btw, "why" questions are always a bit ill-defined, it's like asking for some kind of narrative to satisfy the questioner. "well in the beginning there was the big-bang..." So anyway I'm quite impressed at these papers.

You know who the Yann LeCun is, don't you?

Reply on news.ycombinator.com