Hacker News Reader

Comment by p1esk | Hacker News Reader

That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect.

Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, each one adjusting its available weights to perform the same task. During testing, there's no noise - all neurons are back in business and contributing.

View on HN

> Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, each one adjusting its available weights to perform the same task. During testing, there's no noise - all neurons are back in business and contributing.

I was speaking loosely -- dropout is multiplicative Bernoulli noise on the hidden layers.

> That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect.

I'd argue that dropout (and its predecessor in denoising autoencoders) are perfectly valid to see as noise, albeit multiplicative.

You are missing my point - with dropout, you don't have any noise during the operation of the net. The noise we are talking about (circuit noise) is always present.

I am pretty sure that parent meant DNNs love to fit noise. So you two are in strong agreement.

No, we are talking about a random electrical circuit noise in the analog NN hardware. Of course, if the noise is known and fixed, the net could learn to compensate (to a certain extent). The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained.

> The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained.

What would change with temperature that would require retraining? Are you saying the output of an op could depend sensitively on temperature, or that higher temperatures would increase things like thermal or shot noise? Why would the latter require retraining?

Current depends on threshold voltage and carrier mobility, which are temperature dependent.

Reply on news.ycombinator.com