I have a feeling, albeit one not informed by deep knowledge of either field, that there are already-known pure mathematical results that can cause a <insert your favorite way of saying "paradigm shift"> in machine learning, just "waiting to be discovered" by people in the latter field.

All this talk of maxima and minima reminds me of Morse theory, for instance (and that Wikipedia page is more than what I know about it). [1]

Is there any sense in what I'm saying?

[1]: https://en.wikipedia.org/wiki/Morse_theory