I've been on my soapbox for a while that introductory machine learning courses should cover the basics of optimization. Even just basic KKT conditions could add a lot.
Full respect to this piece for including the phrase “the simplest possible thing”.