JunaidB

I think this is a concise and good overview of ML model evaluation. I was recently thinking along the same lines and posted it on my blog here: https://scienceofdata.org/2020/02/23/ml-classifier-evaluatio...

You have nice graphs and cover regression models too. I use mostly these when evaluating models I've built, I also use the Kappa statistic which can be a useful summary.

This seems like an excellent review. I'll check it out. Thanks very much!

Your point about convergence to the exact minima over training loss not guaranteeing the best generalization loss reminds me of the point made in this lecture here https://www.youtube.com/watch?v=k3AiUhwHQ28.

You also made an interesting comment about work not catching on outside of the optimization community - can you recommend some resources or websites to follow in order to see what the optimization community is working on? I've developed an interest in the area but don't really know where to go for "up to date" information.

Thank you for taking the time to write a thorough and considerate response. I have been working through the Engineering Optimization Methods and Applications by Ravindran, Ragsdell and Reklaitis so far but I will spend some time in the coming few weeks with Nocedal and Wright in accordance with your recommendation.

I intend to write more about what I learn in this area and I'd be honoured if you would contribute like you did here with your comments/ corrections and suggestions! Thank you for the help and reference, I will definitely be following up.

That's a great point and to be honest I could have been a lot tighter with the terminology. Good advice to take on board for next time - thanks!

Your point about combining optimisation techniques is interesting and I'd love to learn about it a little more. When you say "As such, most good, fast optimization algorithms based on differentiable functions use the steepest descent direction as a metric, or fallback, to guarantee convergence and then use a different direction, most likely a truncated-Newton method, to converge quickly", does this mean that both algorithms are being used together? So first steepest descent is run for a few iterations and then the truncated-Newton method takes over?

If you have some resources where I could read up on this it would be much appreciated!

This is a really important point and I wish I'd mentioned it. The computational considerations (as you've said) make the classic Gradient Descent method infeasible in practice. Therefore we resort to stochastic estimates or Quasi Newton approaches (which I'm still looking into).

My main objective was to highlight is that given that we are performing the classic gradient descent, the gradient will yield the greatest reduction in the function value. Essentially it was a point to highlight the underlying calculus. Wayne Winston in his book Operations Research: Applications and Algorithms has an interesting passage where he discusses the gradient being the direction of maximum increase (he was looking as steepest ascent).

Not rude at all! Thanks for the question. My background is in Statistics not Machine learning (that came later) so I covered these topics without reference to ML applications. I suppose I never learned to connect these ideas when I was learning about ML, it was only when I went back to my old material I realised how these ideas relate. I agree with your point about teacher material, when I was learning about ML it was separated from raw Calculus.

A short answer to your question - yes I took those courses before I started looking at the more integrated material.

Thank you for taking the time to look through the post. When I learned this formally it was introduced as steepest descent (Cauchy's variant). Like yourself, it didn't become concrete to me until I looked at the surface plots. I concur with your point that it's interesting to see the different paths people take toward learning the same thing.