Hacker News Reader

Comment by cl3misch | Hacker News Reader

I find the nomenclature in this article a bit weird.

> Another disadvantage of backpropagation is its tendency to become stuck in the local minima of the loss function. Mathematically, the goal in training a model is converging on the global minimum, the point in the loss function where the model has optimized its ability to make predictions.

"Backpropagation" is the method how to compute the gradient of the weights with respect to a loss function. But the article repeatedly uses the term as if it was the whole optimization algorithm, running into local minima.

View on HN

Wikipedia: "The term backpropagation strictly refers only to the algorithm for computing the gradient, not how the gradient is used; however, the term is often used loosely to refer to the entire learning algorithm, including how the gradient is used, such as by stochastic gradient descent."

-- Meaning follows usage.

https://en.wikipedia.org/wiki/Backpropagation

This is wrong, stupid and extremely confusing.

Could you provide a definition correct, smart and sorta helpful?

As someone who uses this in their day job, I have no problem using loose terminology to describe a well known procedure. Most of the time when I am referring to the act of optimization, it doesn’t matter exactly what method I am using, and I can use backprop as a stand in. If I’m talking about the technical details of my work, I will state the specific optimization strategy. Everyone on my team does similar things, and no one is confused or misled. Use rigorous language when necessary, and use colloquial language when appropriate.

> Use rigorous language when necessary, and use colloquial language when appropriate.

Do you think a peer-reviewd publication is formal enough to warrant precise language? These publications are not only read by specialists in the field. I use automatic differentiation in my daily work, but I'm not familiar with machine learning. Thus I am very confused when "backpropagation" is used to mean an optimization algorithm.

EDIT: It is as if physicists used the term "special relativity" to talk about "quantum mechanics" because, after all, quantum mechanics happens in Lorentzian spacetime. Now for specialists of quantum physics it may make sense, since they are using "special relativity" to distinguish it from fancier quantum theories that combine field theory with GR. But for normal people it would be certainly misleading. Using "backpropagation" to include optimization has the same feeling.

QM doesn't happen in lorentzian spacetime, QFT does though

yes, sorry, I meant quantum field theory

Don't worry about it, it was close enough to get your point across.

"Gradient descent" is more accurate and equally "loose".

Some people use backpropagation and gradient decent interchangeably. It's really confusing though

I wouldn’t say it’s confusing—it’s wrong and suggests at the least sloppiness in terminology

> sloppiness in terminology

venturebeat.com

Not the worst, but we're not talking Nature or Spectrum here.

of course it is wrong. One nice thing about math is that thing are defined precisely and back propagation and, say, SGD or Adam are different things

That's bad. There are gradient descent techniques that are feed-forward, and understand the domains where those are appropriate and the domains where backprop is appropriate is I think important, especially as you try to do things like mix machine learning techniques with other differentiable programming strategies.

Deep learning terminology is often loose like that, see "softmax loss" or "deconvolution layer" etc.

Reply on news.ycombinator.com