Hacker News Reader

Comment by abeppu | Hacker News Reader

I kept expecting terms like "ensemble" and "feature bagging" to pop up, but they didn't. How are others mentally mapping these concept back to ML?

View on HN

Jeff has a whole section on machine intelligence in his upcoming book and discusses this in part 2 of the teaser podcast [0].

[0] https://www.youtube.com/watch?v=zTRZL8dqHoo

Also "dropout".

Yeah, dropout is conspicuously missing. For those who haven't heard: Dropout zeros out random activations, creating a random subnetwork to make a prediction. Each training step updates a different random subnetwork. (...Keeping in mind that two random subnetworks can share lots of weights...) Then at inference time, you use the full network, which now acts as an ensemble of all of the subnetworks.

You could view it as 'thousand brains' with lots of weight sharing for efficiency+regularization.

Much like backprop, I'm not sure dropout can be mapped back to the way an actual brain works.

Wait, why can't you? Backprop is basically adjusting weights to get the model closer to making correct predictions. Isn't that basically how the brain does it? (Though I'm sure the way the brain adjusts its weights uses a different approximation of differentiation)

I think the main issues are:

- there's not a "correct" training prediction

- there's not a "final layer"

Backprop is not simply about adjusting the weights (as a matter of fact, you can argue that any training method is about adjusting the weights).

It's about computing the amount by which you adjust the weight.

And unless there's been a major development in neuroscience that I'm not aware of, backprop is not the way the brain does it.

Backprop requires global communication but biological brain works with local communication in neural assemblies, where signals spread in a way somewhat similar to diffusion.

There is no global “teaching signal” or “delta rule” error correction. Learning via reward and punishment is a wrong level of abstraction for fundamental cognitive tasks like visual object recognition or “parsing” auditory signal.

Sometime people mention dopamine as a kind of reinforcement signal but it operates on completely different time scale, orders of magnitude slower than any iterative optimization model would require.

And the energy and time spent on iterative optimization in ANNs is not available to living organisms with constrained resources.

If you’re interested in authoritative opinion on what kind of learning is biologically plausible see e.g. prof. Edmund Rolls recent book called “Brain Computations”.

Reply on news.ycombinator.com