Hacker News Reader

Comment by deepnet | Hacker News Reader

You need an error signal - a target value is compared with the networks prediction. That error is carefully assigned proportionally to the network weights that contributed to it and the weights adjusted a small amount in that direction. This is repeated many times.

Backpropagation suffers from vanishing gradients on very deep neural nets.

Recurrent Neural Nets can be very deep in time.

Or the weights could be evolved using Genetic Programming.

View on HN

> Backpropagation suffers from vanishing gradients on very deep neural nets.

Especially when using saturating functions (tanh/sigmoid)

> Or the weights could be evolved using Genetic Programming

GA, not GP http://en.wikipedia.org/wiki/Genetic_algorithm

> Or the weights could be evolved using Genetic Programming.

Some algorithms, such as NEAT[0], use a genetic algorithm to describe not only the weights on edges in the network, but also the shape of the network itself - e.g., instead of every node of one layer connected to every node of the next, only certain connections are made.

0. http://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_to...

It would be interesting to occasionally train the generated C against a compiler.

In "Learning to Execute" by Zaremba & Sutskever http://arxiv.org/abs/1410.4615 An RNN learns snippets of python

Their next paper is "Reinforcement Learning Neural Turing Machines" http://arxiv.org/abs/1505.00521 based on Graves "Neural Turing Machines" http://arxiv.org/abs/1410.5401, which attempts to infer algorithms from the result.

In a lost BBC interview from 1951 Turing reputedly spoke of evolving cpu bitmasks for computation.

It would also be ideal to use a higher level interpreted language, and have it try to generate one page scripts rather than giant mega projects like linux.

Reply on news.ycombinator.com