I'm surprised that genetic algorithms on resource-bounded general purpose virtual machines have not surpassed neural nets in their problem solving ability.
Is the computational advantage of neural nets on GPUs so dramatic?
Is the computational advantage of neural nets on GPUs so dramatic?
When you don't have or can't use a gradient, GAs become the go-to tool, for optimizations (if only because you don't have much in the way of other options).
That said, I use GAs for optimization and find them to be an incredibly useful tool for various optimization tasks - just generally not for learning weights of NNs, where a GPU + backprop generally reigns supreme (or at least works well enough/much faster than a GA).
I suspect that with some more engineering and attention from people doing ML stuff, GA-style algorithms can be made just as memory and space efficient as gradient methods, while giving better results and being more widely applicable.
Here's a post on this: http://pchiusano.github.io/2020-10-12/learning-without-a-gra...
Here's a post on this: http://pchiusano.github.io/2020-10-12/learning-without-a-gra...
Traditional GA has a practical problem for training large models - keeping a large population of model weights in memory is not feasible. Like if you had to keep 1000 variations of the GPT-3 model weights in memory for training that's no good. Though people have ideas for how to solve for that as well (again, see the post I linked).
parallelism like that isn't possible with GA