Hacker News Reader

Comment by blamestross | Hacker News Reader

The answer is always "No it doesn't need all those nodes", but we do it that way because it makes doing the math easier. I bet they just removed a bunch of edge weights to nodes, but used the same matrices for calculating output.

I wish arbitrary topology networks were scalable (I love NEAT) but bipartide graphs crunch good in GPUs.

View on HN

I'm quite surprised so much was retained; many presentations on convolutional networks suggest adding a lot of features, but prune most in a post learning step. Supposedly lots of features, which are initialized randomly, just raise the probability of being close to the real answer (the concept being learned). Once it's found, most of features typically go unused or are reduntant.

Can you elaborate what you mean by "edge weights to nodes, but used the same matrices"?

The network retains the same topology, and therefore the same matrix dimensions, but some elements are set to zero, removing their contribution.

Relevant excerpt from https://news.ycombinator.com/item?id=38656172

> ...if we meet the shape 512x512 (512 is a power of 2, therefore a very "round" number in computer science), then maybe some kernel will be very fast; but then on a 512x511matrix, the same kernel may need to add some padding first to transform it into a round 512x512 matrix with zeros at the end of each row. Adding those zeros means shifting all rows, which is a very costly operation.

Reply on news.ycombinator.com