Genetic Algorithms for Training Deep Neural Networks for Reinforcement Learning
arxiv.org
arxiv.org
Through the history of deep learning, the frontier has been networks that can be trained to some feedback of whether they're working or not in ~6-8 days. Within a few years there will almost certainly be ASICs that allow us to train hundreds of thousands or millions of networks in several weeks, relative to several weeks for one of the same network in 2017. This will let us start exploring things like evolving networks seriously.
Fundamentally, we know neural networks can instantiate general intelligence, and we know genetic search is capable of finding the right neural networks. There are big differences between the CS and biological versions of each, but it's striking that the big breakthrough in "AI" was deep neural networks and not anything else.
When I think about the difference between AlphaZero and human intelligence, I don't think it's "more intelligence." AlphaZero seems perfectly intelligent to me: I think the difference is more about the selective pressures that produced us. AlphaZero is a reflection of its environment and the process by which it developed in that environment. I would be shocked the future of deep learning continues to be hand-design from human intuition.
Edit: looking at the link, I want to caveat the above by saying this paper may or may not be "it," but directionally I think the idea is underrated and erroneously out of favor now as neural networks were in 2009. Ken Stanley (coauthor on the linked paper) in particular has been hung up on one particular approach, NEAT, since like 2002 that is kind of interesting but definitely not the end-all.
This isn't at all the case! We know neuronal networks, of the human brain variety specifically, can host this vague thing we're calling general intelligence.
It's not at all clear that artificial neural networks of the deep learning variety can do everything a neuronal network can do.
My feeling is that since shallow networks can be made to have equivalent accuracy to deep networks, that the real challenge isn't topology but training. Hobbyists have access to so much processing power with GPUs now that they can explore techniques that weren't practical for experts 20 years ago. So we may see training speed increase by a few orders of magnitude using techniques besides gradient descent (maybe quantum computing someday, who knows).
The big question though is how to combine networks into hierarchies so that the number of behaviors that can be learned is no longer limited (since pattern recognition is largely a solved problem). I think the way GAs fit in is that they make it much easier to understand and build simple NNs, and possibly train hierarchies or discover topologies that aren't immediately obvious.
> My feeling is that since shallow networks can be made to have equivalent accuracy to deep networks, that the real challenge isn't topology but training.
This is not really true though... even very shallow neural networks can be universal function approximators in a trivial sense because they can be lookup tables, but they are really not expressive enough to generalize well and lack a lot of the expressivity of deep networks.
You've got it flipped. If anything, shallow neural networks (of an equivalent number of parameters) are more "expressive" than deep networks, BUT that expressivity just makes them overfit. This is the bias vs. variance tradeoff. If anything, deep networks encode our prior belief that there is a hierarchy of features / a compressed representation, which limits the model that is learned, to a model conforming to those priors.
Disclosure: I contributed to the linked work.
To do simple tasks, you don’t even need neural networks, in short. If you want to do complex tasks, training the neural net isn’t the problem, but a NN can indeed represent more than say a linear policy. The issue is getting the NN to do the complex thing correctly: it’s more of an exploration problem than function approximation problem.
Ssh! We've got a gold rush to maintain.
In a sense, topology is training. One reason that DNNs outperform shallow networks for so many problems is that presumably their topology captures structure inherent in the world that would otherwise have to be discovered by a simpler network.
Also previously on HN http://www.visiondummy.com/2014/04/curse-dimensionality-affe...
More seriously, thinking about metaheuristic, which approaches at which scale, i.e. machine learning architecture, that's the future.
Stochastic hill climbing as a baseline method for evaluating genetic algorithms: http://papers.nips.cc/paper/1172-stochastic-hillclimbing-as-...
When will a genetic algorithm outperform hill climbing? http://web.cecs.pdx.edu/~mm/nips93.pdf
A GA being competitive with modern gradient based methods is very surprising to me.
Thing about evolutionary networks is that they can evolve topology, which usually is just a thing decided by people implementing NNs (and not guaranteed to be anywhere near most optimal). I don't see why given enough time GAs (or EAs) could not outperform simple backpropagation based on differentiating simple cost functions.
It's already happened. You are it.
Looking at current methods of optimizing neural networks we cannot know if we are far from the upper bound or if we've already reached it... R&D effort in this area could be a dead end.
The person I was replying to was also advocating the use of GAs to solve the antenna problem, which don't use gradient information either.
https://www.amazon.com/product-reviews/1558607838
People need to recognize that this is not a new idea. It may have been groundbreaking in the early 2000s, but Genetic Algorithms + Neural Nets are a relatively old strategy at this point.
Definitely read the book if you want a bit of a throwback. When cutting-edge AI research was about beating players on Yahoo-games Checkers with new techniques.
It's a shown/kind of proven result that deep neural networks don't fall into local minima in the very high dimensional parameter space.
GANs and reinforcement learning are different. Research on getting those to converge to good local minima is still much more in its infancy. I don't particularly consider those just a "neural network", but sorry, I should have been more clear.
But even vanilla supervised nets suffer from local minima. Anyone who's played with them has encountered it. Here you can mess around with a neural net live in the browser and it very easily gets stuck if you try more than 3 layers (especially try the spiral dataset): http://playground.tensorflow.org/
Check any of the literature on this subject: https://arxiv.org/abs/1611.06310v2
https://arxiv.org/abs/1406.2572
Local minima are something that people thought was gonna be a problem, especially back in the 2000s. They played around with small neural nets on toy examples such as yours, and thought it was intractable. It's the entire reason why neural nets fell out of the fashion in the early 2000s, and people moved towards techniques like SVM.
These toy examples don't generalize to high dimensions, and if you take a look at the literature, you'll see that the consensus agrees with my statement.
Maybe with a billion neurons, just by random chance some of them would correspond to the correct algorithm and get reinforced by backprop. But very few NNs have layers larger than a thousand neurons. Because the cost of layers that big grows quadratically. And the chance of random weights finding the solution decreases exponentially.
One of the biggest reasons things like stochastic gradient descent, and dropout are used is because they break local minimas.
These are not just theoretical results. They're theory papers trying to explain the empirical result of why neural nets don't get stuck at local minima.
> Given that deep networks are highly nonlinear systems optimized by local gradient methods, why do they not seem to be affected by bad local minima?
And other such results.
As I said above, neural nets are obviously able to get stuck in local minima in toy examples. If you read my above comment, you'll see that that has no bearing on my initial statement.
Dropout's main motivation is not to break local minima. It's to achieve better generalization. If it were the case that it was meant to break bad minima, we'd have better training loss upon adding dropout, which is obviously not true.
As for SGD, we used to think that it was mainly for computational purposes. That is, we're unable to batch our entire training set at once, so we have to split into mini batches.
Modern theory states more that SGD is good for avoiding sharp minima, as well as some other desirable properties.
I'm not sure you're really reading my comments thoroughly nor checking out the links, so if you're actually interested in understanding what's really going on, please do some proper research on the topic.
I do know that we define the cost function / the goal of it, but from using evolutionary Technic to build the network, there is only x layers left to add to create any net.