Hacker News Reader

Comment by dave_sullivan | Hacker News Reader

Call me crazy, but isn’t this “single biological neuron” actually 2 locally connected layers with a field width of 2 and unshared weights with a third fully connected layer at the end? With a relu nonlinearity?

I’m not surprised this does well on MNIST and I’m not sure it breaks with present research directions in deep learning. This network could be built pretty easily in torch or tensorflow.

View on HN

Yes, I stopped reading as soon as I found they used gradient descent. Our brains does not uses gradient descent to learn. If you use any sufficient complex model, and fit MNIST by defining a loss and minimizing it directly, it will very likely work. Even simple linear model of logistic regression on 784 pixels gives good accuracy of around 90%.

A simpler description would be a sparsely connected network, with a tree shaped connectivity pattern.

There is a lot of research potential waiting to be discovered (or maybe rediscovered) with sparsity in neural networks, so I am a fan of this area of research. I would even say that sparsity is playing a much bigger role than we realize in the success of many neural architectures that are popular today.

By being simple does not mean that it is not insightful

I’m not seeing anything insightful though. They are taking a fairly standard architecture, claiming it’s something different, and then claiming some connection to biological neurons.

As I read it, this is an example of people publishing a minor tweak in the algorithm and then claiming it’s a new biologically inspired abstraction. I guess I just want more papers about transformers.

Reply on news.ycombinator.com