Link Between Deep Neural Networks and the Nature of the Universe
technologyreview.com
technologyreview.com
Often these threshold integration neurons are just approximating a simpler Boolean function so it really comes down to digital logic.
Techniques for pruning networks can be found here:
The problem becomes even more difficult if there is no clear distinctions between layers in the network, as in the brain.
You're also assuming that holographic networks are somehow necessary for most of the tasks we require. The human brain, while having billions of neurons is still quite sparsely connected.
One of the researches is Max Tegmark, author of Our Mathematical Universe. In short, he suggests every theory becomes it's own universe; physical laws are representations of patterns in some kind of calculation or proof.
As he explains, although every possible theory is a universe, "most" complex theories can be reduced to simple ones. So it seems plausible that those simple patterns are more common and more likely to be found in a random (mathematical or physical) universe.
This means that neural networks are likely to work with those common patterns. One might say it because that's closer to the nature of physical laws. But more directly, that's because of the nature of mathematical laws.
This applies especially well to string theory which is just layers of pointless mathematics, each layer trying to make up for obvious deficiencies of lower layers.
The answer is that neural networks are a somewhat crude approximation of Solomonoff induction. Solomonoff induction is an ideal, perfect machine learning method. It only assumes that the universe is a random computer program, and tries to infer which one.
Additionally, a paper that has everyone excited about deep connections between the mathematical analysis of physical systems and the hierarchical feature learning paradigm speaks of the connection in terms of the Renormalization Group [2].
Regardless of it's practical utility, the philosophical implications do tickle the intellect. On a dreamy note, I wonder if it would be possible to draw Category-theoretic parallels between some physical theories and statistical learning theory. There is so much to learn, and I am trying my best to teach myself (on the side) the mathematics that they don't teach in my CS grad school. So much to learn, such little time. :)
Is that true? After reading Jeff Hawkings - "On intelligence" book (2004), for me it's pretty much clear why they are so successful on that set of tasks. Despite of HTM model built by Jeff being kind of different from DNN, the idea and origin of brain and consciousness described pretty well. All modern advances in neuroscience just prove how strong the theory in this book is.
What's less clear is why HTM has not been successful on any tasks since 2004. I remember a few years ago they entered Kaggle competition to detect anomalies in EEG. Their team finished in ~150th place (out of 500). Even though anomaly detection in time series data was supposed to be their strongest feature.