This is essentially repeating your quote, but an "aha moment" clicked for me when I read that what successful neural networks are basically doing is such good feature learning that the problem can be solved by a simple linear model in the end.

E.g. if you have an N-layer neural network, N-1 layers are doing feature learning, and the Nth layer is a simple {logistic, multinomial/softmax, gaussian, poisson, ...} model