Take a look at https://ocobook.cs.princeton.edu/OCObook.pdf, and in particular section 1.2. The online shortest path problem is a good example of an online optimization problem that's not related to machine learning.