Hacker News Reader

Comment by martindbp | Hacker News Reader

You have to remember what came before 2012: SVMs, Random Forests etc, absolutely nothing like the brain (yes, NNs are old, but 2012 was the start of the deep learning revolution). With this frame of reference, the brain and neural networks are both a kind of Connectionism with similar properties, and I think it makes perfect sense to liken them with each other, draw inspiration from one and apply it to the other.

View on HN

It is also odd to see such a weak argument as the brain-to-body mass ratio being used, as here: https://youtu.be/YD-9NG1Ke5Y?t=593

If this metric were truly indicative, what should we make of the remarkable ratios found in small birds (1:12), tree shrews (1:10), or even small ants (1:7)?

https://en.wikipedia.org/wiki/Brain%E2%80%93body_mass_ratio

> what should we make of the remarkable ratios found…

We also can’t implement those creatures’ control systems in silicon, so they too are doing things we can learn from?

What came before was regression. Which is to this day no 1 method if we want something interpretable, especially if we know which functions our variables follow. And self attention is very similar to correlation matrix. In a way neural networks are just bunch of regression models stacked on top of each other with some normalization and nonlinearity between them. It's cool however how closely it resembles biology.

sorry, but i think neural-networks came way before 2012, notably the works of rumelhart, mccleland etc. see the 2 volume "parallel distributed processing" to read almost all about it.

the book(s): https://direct.mit.edu/books/monograph/4424/Parallel-Distrib...

a-talk: https://www.youtube.com/watch?v=yQbJNEhgYUw

Jets and Sharks!

https://github.com/acmiceli/IACModel

I raise you Warren McCulloch in 1962: https://www.youtube.com/watch?v=wawMjJUCMVw

I knew someone would bring it up, which is why I added "(yes, NNs are old, but 2012 was the start of the deep learning revolution)"

2012 was when the revolutionaries stormed the bastille and overthrew the old guard. But I say it was 2006 when the revolution started, when the manifesto was published: deep NNs can be trained end-to-end, learning their own features [1]. I think this is when "Deep Learning" became a term of art, and the paper has 24k citations. (Interestingly in a talk a Vector Hinton gave two weeks ago he said his paper on deep learning at NIPS 2006 was rejected because they already had one.)

[1] G. E. Hinton and R. R. Salakhutdinov, 2006, Science, Reducing the Dimensionality of Data with Neural Networks

Sorry, but they just called it "neuron" to sound nicer.

Neural Networks are 200 years old (Legendre and Gauss defined Feed forward neural networks). Deep learning. The real difference between traditional ones and deep learning is a hierarchy of layers (hidden layers) which do different things to accomplish a goal. Even the concept of training is to provide weights on the neural network and there are many algorithms to do refinement, optimization and the network design.

Gauss did not define feed forward neural networks, it all stems from a tweet of a very confused person.

I mean, sure, you can model a simple linear regression fitted via Least Squares (pretty much what they did 200 years ago) with a one hidden layer feed-fwd Neural Network, but the theorical framework for NNs is quite different.

For Least Squares, you do not even use a hidden layer. Just a single dense layer from input directly to output. You also do not use an activation function (or use the identity activation function). That is, everything that makes neural networks special.

Reply on news.ycombinator.com