Stanford Stats 385: Theories of Deep Learning
stats385.github.io
stats385.github.io
Let me see if I can state the problem: Neural Networks are non-linear because of their activation functions. You need a differentiable function in order to take the derivative so you can back-prop the error, more or less.
The consequence of the non-linearity is that you can't do some kind of short hand calculation to figure out what the network will do. You have to crank through the network to see the result. There is no economy of thought. That is to say, there is no theory.
I am excited that they are working on it, but I would love to have a summary or overview of how they approach what I consider to be the basic problem.
To clarify: linear functions would still be differentiable without an activation function. The problem is that the composition of linear functions is just another linear function, so you gain nothing by having multiple layers; you might as well just do some kind of linear regression or classification. Activation functions introduce nonlinearity so that deep learning methods can hopefully learn things linear methods can't.
I would just like to clarify: deep learning methods definitely DO learn things that linear methods can't.
Using linear functions, no matter how many layers, essentially boils down to the "perceptron" architecture. You can Google it and they will be mostly talking about it's limitations (for instance, it's famously unable to learn XOR, Google "perceptron XOR", essentially the issue is that XOR is not linearly separable, so it can only be expressed as a nonlinear function).
You can give a simple proof for this. If you examine what a neural network layer (without activation function) does, in matrix terms. You take the input vector X (1xn), the layer weights W (nxm), and the output vector O (1xm).
Then computing the layer output is simply O = WX.
Now we can envision what happens with two layers. W (mxn) and V(mxn). The output of running two layers then becomes:
O = V(WX)
However, matrix multiplication is associative:
O = V(WX) = (VW)X, and VW is just a matrix
So for every 2 layer linear neural network there is a 1 layer neural network that gives the exact same result. So there's no reason to have 2 layers.
A famous result proving this is the "deep" networks (with activation funcions) are universal function approximators (explained here: [1]). Note that this "deep" should be understood in the 1995 meaning of deep neural networks, which is essentially "at least 2 layers", and usually "exactly 2 layers", not the 2010+ one where one means 10 to 200 layer deep networks.
Furthermore, a non-linear 2-layer node can map any arbitrary function so the argument for more nodes is learnability.
https://en.wikipedia.org/wiki/Universal_approximation_theore...
That means there is no 1-layer perceptron that can learn XOR, and there is no multilayer perceptron that can learn XOR.
I agree with every second sentence: Given that every perceptron layer does a linear transformation of the input.<- True There is no linear function that separates XOR, and that also means there is no sequence of linear functions that separates XOR. <- False That means there is no 1-layer perceptron that can learn XOR, <- True and there is no multilayer perceptron that can learn XOR.<-False
Please look it up. Here are a few links:
http://toritris.weebly.com/perceptron-5-xor-how--why-neurons...
The graph in slide 3 of this link helps explain it: http://www.di.unito.it/~cancelli/retineu11_12/FNN.pdf
http://www.mind.ilstu.edu/curriculum/artificial_neural_net/x...
But that makes multilayer perceptrons different from ordinary perceptrons in more than just the multilayer part, which is very confusing.
Bottom-up is the preferred solution because you can derive everything from earlier principles, math is always bottoms up. Physics tries to be but occasionally fails.
Top-down is investigatory, you have something that exists and you want to understand it. Chemistry, biology, and genetics are excellent examples of systems that we can't derive from first principles that we've done well with using the top-down approach.
Charting the space of the meta-problem (this config --> this behavior) is a valuable first step in better understanding these structures even if we can never derive their behavior.
There's a spectrum between "just keep trying tons of crap and see what works" and "do this simple, well-understood calculation to see exactly what will work." Is it not obvious why it's nicer to be on the latter end of the spectrum than the former?
Also, I'll mention that you're being too pessimistic about what theory can accomplish. The learning problem is much more constrained than P = NP.
The use, for example, of a training set of N examples, drawn iid, and evaluation on samples drawn from the same distribution imposes a lot of structure. I don't know if you're familiar with VC theory, but it's an example of the kind of "surprising" guarantees that can be derived in this setting. Other general examples are weak learning, the bias/variance tradeoff, and (in SVMs) the notion of large margin classifiers.
An applicable "theory" of design is what separates engineering from just mucking around.
The way I understand experimental physics, its purpose is either (a) to make observations on which new theories can be built or (b) to experimentally verify, or falsify, existing theories.
I don't think it's quite like throwing stuff at a wall and looking to see what sticks.
Sounds more like alchemy than experimental physics to me.
Well, the reason why we have computers today in the first place is because Turing (and Church, btw) solved a long-standing problem in mathematical logic. That was a theoretical problem -Gödel's incompleteness theorem- and his solution was also purely theoretical: Turing machines (Church's solution was the lambda calculus; btw, both solutions basically led to a new problem, the halting problem. I digress).
Without these advances in theory, do you think we could have ever invented computers? If you do, you shouldn't. Instead, you should try to imagine trying to invent a computer by enumerating math theorems without a computer.
Today of course, we have computers. However, we also still have problems that cannot be solved by brute-force enumeration of solutions- just like pre-computer mathematicians had problems they couldn't solve just by enumerating solutions without a computer. For this type of problem, we have to use our brains and come up with theoretical solutions in order to make progress.
It may be out of our league, or we may just need a genius to help figure it out.
Calculus and Newtonian Mechanics and Relativity were probably out of our league before they weren't.
There are many parameters (layers, nodes, connectivity, drop off, number of trials etc). Right now they are guessed at by trial and error. The thing preventing you from saying, "Oh, just reduce the layers by 1, and add more nodes" is the non-linearity.
[1] https://twitter.com/stats385
[2] https://www.researchgate.net/project/Theories-of-Deep-Learni...
How on earth does their comment imply that?
I now have no idea why it was down-voted.
However, Quora seems to think you may be able to sneak in, at least sometimes:
https://www.quora.com/What-does-Stanford-do-about-non-Stanfo...