Start with the MIT linear algebra course (18.06) by Gilbert Strang and Stanford course on linear dynamical systems (EE263) by Stephen Boyd. Then move on to Boyd's course on convex optimization (EE364). Lectures for all of these are on youtube.

Do not try to read any books on "machine learning" (most of which are a total mess) before you have this background or you will just end up hopelessly confused.