Crucially it will tend to find the simplest such representation that still solves the problem. This is why ultimately the model is only sufficient to solve problems that it was trained to solve.
Simplest in terms of optimization, by the way.
Simplest in terms of optimization, by the way.
I get it find solution that are easy for SGD or Adam optimizer to find.
But why would such solution be less simple than other ?
I think the comment you're replying to means exactly what you're saying, which is that it will find solutions which are "easy" to find for the optimizer, and therefore solutions which are simple to achieve through the convergence of some optimizer.